System and methods for intraoperative guidance feedback

ABSTRACT

Systems and methods for surgical guidance and image registration are provided, in which three-dimensional image data associated with an object or patient is registered to topological image data obtained using a surface topology imaging device. The surface topology imaging device may be rigidly attached to an optical position measurement system that also tracks fiducial markers on a movable instrument. The instrument may be registered to the topological image data, such that the topological image data and the movable instrument are registered to the three-dimensional image data. The three-dimensional image data may be CT or MRI data associated with a patient. The system may also co-register images pertaining to a surgical plan with the three-dimensional image data. In another aspect, the surface topology imaging device may be configured to directly track fiducial markers on a movable instrument. The fiducial markers may be tracked according to surface texture.

CROSS-REFERENCE TO RELATED APPLICATION

This application is a continuation of U.S. Ser. No. 13/664,613, titled“SYSTEM AND METHODS FOR INTRAOPERATIVE GUIDANCE FEEDBACK” filed on Oct.31, 2012, the entire contents of which are incorporated herein byreference, which claims priority to PCT Patent Application No.PCT/CA2011/050257, titled “SYSTEM AND METHODS FOR INTRAOPERATIVEGUIDANCE FEEDBACK” and filed on Apr. 28, 2011, the entire contents ofwhich are incorporated herein by reference, which claims priority toU.S. Provisional Application No. 61/328,679, titled “SYSTEM AND METHODSFOR INTRAOPERATIVE GUIDANCE FEEDBACK” and filed on Apr. 28, 2010, theentire contents of which are incorporated herein by reference.

BACKGROUND

The present disclosure relates generally to surgical guidance.

Image-guided target tracking and surgical guidance is a method forlocating a specific target within three-dimensional (3D) space. Thistechnique is routinely used in medical procedures to locate an object inthe human body, such as the spine, brain or other organ structures,during surgery.

One approach to a guided surgical intervention includes the use offiducial markers that are attached to the body with a clamp, anadhesive, or through other means. Generally, these fiducial markers arealigned to a 3D representation of the body, which may be acquired bydifferent imaging modalities. This 3D representation, usually acquiredbefore surgery, may include a specific region, such as a vertebralcolumn, to a scan of the entire body. Within this 3D representation,areas of interest are located and matched to the fiducial markers in thereal surgical space. This results in a coordinate system transform thatmaps the relative position of the region of interest to the location ofthe fiducial markers to provide visual feedback to the clinician duringsurgery. The surgeon can then use this information to facilitateguidance to a specific location in the body that is related to theregion of interest in the image.

Optical-based surgical navigation has been used for the past decade toguide spinal surgeries and, in particular, placement of screws in thespine. These systems are based on two cameras that detect light that iseither emitted (mounted with LEDs) as disclosed in U.S. Pat. No.5,921,992, or passively reflected from surgical tools and probes asdisclosed in U.S. Pat. No. 6,061,644. Using the signal detected by thecameras combined with the knowledge of the dimensions of the navigationprobes, a computer workstation is able to precisely determine where thetip of the surgical instrument lies.

U.S. Pat. Nos. 5,531,520 and 5,999,840 provide a system that utilizes aplane of laser light and a video camera to obtain three-dimensionalmeasurements of the patient's skin, where the system employs the“structured light” method of obtaining the desired measurements forregistration of 3D pre-operative image data. Prior to a surgicalprocedure, pre-operative MRI or CT data is first obtained. Subsequently,in an operative setting, the patient is scanned by a laser rangescanner. The pre-operative MRI or CT scan is automatically registered topatient skin surface obtained by the laser range scanner, providing atransformation from MRI/CT to patient. The position and orientation of avideo camera relative to the patient is determined by matching videoimages of the laser points on an object to the actual 3D laser data.This provides a transformation from patient to video camera. Theregistered anatomy data is displayed in enhanced visualization to “see”inside the patient.

The registration process taught by U.S. Pat. No. 5,999,840 alsodiscloses the tracking of surgical instruments and probes. A probe istracked by a separate probe tracking system, in which dedicated probetracking cameras are employed to track a probe. The tracked probe datais then registered to the three-dimensional skin surface data using acalibration process. Thereafter, the data registration between the probeand the skin surface is used to provide visualization information to thesurgeon.

In order to track the probe, a calibration procedure is needed toregister the reference frame of the probe tracking system to that of theoptical surface measurement system. This calibration process involvesthe measurement of a calibration object. The process requires that theprobe tracking reference frame be fixed relative to the optical surfacemeasurement system to maintain calibration, such that the opticalsurface measurement system cannot be moved relative to the probetracking reference frame intraoperatively. This requirement canconstrain surgical workflow and cause a need for inter-operativere-calibration of the system.

SUMMARY

Three-dimensional image data associated with an object or patient isregistered to topological image data obtained using a surface topologyimaging device. The surface topology imaging device may be rigidlyattached to an optical position measurement system that also tracksfiducial markers on a movable instrument. The instrument may beregistered to the topological image data, such that the topologicalimage data and the movable instrument are registered to thethree-dimensional image data. The three-dimensional image data may be CTor MRI data associated with a patient. The system may also co-registerimages pertaining to a surgical plan with the three-dimensional imagedata. In another aspect, the surface topology imaging device may beconfigured to directly track fiducial markers on a movable instrument.The fiducial markers may be tracked according to surface texture.Example implementations described herein provide a system for providingsurgical guidance feedback during a surgical procedure.

Accordingly, in one aspect, there is provided surgical guidance systemcomprising: a storage medium for storing pre-operative image dataassociated with a patient; an integrated surface topology imaging andoptical position measurement device comprising: an optical projectiondevice for projecting optical radiation onto an exposed surface of thepatient, such that backscattered optical radiation is suitable foroptical surface topology detection; an optical source having awavelength selected to illuminate a set of fiducial markers provided ona movable instrument; two or more cameras, wherein at least one of saidtwo or more cameras is configured for imaging the backscattered opticalradiation, and wherein at least two of said two or more cameras areconfigured for imaging the set of fiducial markers when illuminated; asurgical guidance controller operatively connected to said integratedsurface topology imaging and optical position measurement device andsaid storage medium, wherein said surgical guidance controller includesa processor configured to: control said optical projection device toilluminate the exposed surface and obtain, from said at least onecamera, topological image data associated with the exposed surface; andcontrol said optical source to illuminate the set of fiducial markersand obtain, from said two or more cameras, positional image dataassociated with said set of fiducial markers; determine a position andan orientation of said movable instrument relative to said exposedsurface; and register said topological image data, and said position andorientation of said movable instrument to said pre-operative image data;wherein said optical projection device, said optical source, and saidtwo or more cameras are rigidly mounted on a frame, thereby maintaininga fixed calibration of said system without requiring inter-operativerecalibration.

In another aspect, there is provided a method of registering surfacetopological image data to preoperative image data using an integratedsystem comprising a surface topology imaging device and an opticalposition measurement device; the surface topology imaging devicecomprising: an optical projection device for projecting opticalradiation onto an exposed surface of a patient, such that backscatteredoptical radiation is suitable for optical surface topology detection;and one or more first cameras configured for imaging the backscatteredoptical radiation wherein the optical projection device; the opticalposition measurement device comprising: an optical source having awavelength selected to illuminate a set of fiducial markers provided ona movable instrument; two or more second cameras for imaging the set offiducial markers when illuminated; wherein the surface topology imagingdevice and the optical position measurement device are rigidly mountedon a frame; the method comprising: obtaining pre-operative image dataassociated with a patient; obtaining pre-determined calibration data forrelating the coordinate system of the optical position measurementdevice and the coordinate system of the surface topology imaging device;optically scanning the exposed surface of the patient with the opticalprojection device and obtaining, from the one or more first cameras,topological image data associated with the exposed surface; illuminatingthe set of fiducial markers by powering the optical source andobtaining, from the second cameras, optical images of the set offiducial markers; processing the optical images to determine a positionand orientation of the movable instrument relative to the exposedsurface, based on the pre-determined calibration data; and registeringthe topological image data, and the position and orientation of themovable instrument, to the pre-operative image data.

In another aspect, there is provided a surgical guidance systemcomprising a storage medium for storing pre-operative image dataassociated with a patient; a surface topology imaging device comprising:an optical projection device for projecting optical radiation onto anexposed surface of the patient and onto a set of fiducial markersprovided on a movable instrument, such that backscattered opticalradiation is suitable for optical surface topology detection; one ormore cameras configured for imaging the backscattered optical radiation;a surgical guidance controller operatively connected to said surfacetopology imaging device and said storage medium, wherein said surgicalguidance controller includes a processor configured to: control saidoptical projection device to illuminate the exposed surface and the setof fiducial markers and to obtain, from said one or more cameras,topological image data associated with the exposed surface and the setof fiducial markers; and determine a position and an orientation of saidmovable instrument relative to said exposed surface; and register saidtopological image data, and said position and orientation of saidmovable instrument to said pre-operative image data; wherein saidoptical projection device and said one or more cameras are rigidlymounted on a frame, thereby maintaining a fixed calibration of saidsystem without requiring inter-operative recalibration.

In another aspect, there is provided a method of registering atopological image data to pre-operative image data for surgicalguidance, wherein the topological image data is obtained by a surfacetopology imaging device, the method comprising the steps of: storing thepre-operative image data associated with a patient; controlling thesurface topology imaging device to optically scan an exposed surface ofthe patient and to optically scan a set of surface texture basedfiducial markers provided on a movable instrument; recording topologicalimage data associated with the exposed surface and the fiducial markers;processing the topological image data to determine a position andorientation of the movable instrument relative to the exposed surface;and registering the topological image data, and the position andorientation of the movable instrument, to the pre-operative image data.

A further understanding of the functional and advantageous aspects ofthe disclosure can be realized by reference to the following detaileddescription and drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

Embodiments will now be described, by way of example only, withreference to the drawings, in which:

FIG. 1 a is a block diagram illustrating an example implementation ofcomponents in an image-based surgical guidance feedback system,demonstrating flows of system information.

FIG. 1 b is an example schematic of an image-guided surgical guidancesystem in use during spinal surgery.

FIG. 2 illustrates an example implementation of an optical filter with acamera to limit specific wavelengths detected for topology imaging.

FIG. 3( a) is a sample 3D CT image dataset of a torso, preoperativelyacquired from a subject, with X, Y and Z axes indicated.

FIG. 3( b) is a schematic of an example surface topology reconstructionof a spine and corresponding vertebrae segmented from a CT image datasetin a posterior orientation.

FIG. 3( c) is a schematic of the surface topology reconstruction of FIG.3 b in a lateral orientation.

FIG. 3( d) is a schematic of the surface topology reconstruction of FIG.3 b in a cross-sectional orientation.

FIG. 4 illustrates an example implementation of a cone of acceptanceprovided in an example implementation of the image-guided surgicalguidance system of FIG. 1A, and the location of the cone of acceptancerelative to a vertebrae as an example surgical target for implantationof an interventional device.

FIG. 5 is a schematic of the perspective view of an exposed spine ontowhich an example implementation of a binary stripe pattern is projectedby a digital projector for structured light imaging.

FIG. 6( a) illustrates an example implementation of preoperative imageacquisition of a spine of a subject and output including a predeterminedprinciple axis demarcating an implantation trajectory of a surgicalinterventional device.

FIG. 6( b) illustrates an example implementation of intraoperative imageacquisition of a spine of a subject and output including an updatedprinciple axis identified by the surgical guidance feedback system.

FIG. 7 illustrates an example implementation of correction of theprinciple axis of the interventional device due to physical displacementof a position (dashed line) of the vertebrae from the positiondetermined in a preoperative plan (solid line).

FIG. 8 is a flow diagram illustrating an example implementation of amethod of intraoperative surgical guidance.

FIG. 9 is a detailed flow diagram of a method of intraoperative surgicalguidance.

FIG. 10 is a flow diagram of an example implementation of a methodalgorithm for generating a transformation matrix for use in imagedataset registration

FIG. 11 illustrates an example implementation of a method of usingfences to segment individual vertebrae from the spine.

FIG. 12 illustrates an example implementation of updating of apreoperative surgical plan for use in a method of intraoperativesurgical guidance.

FIG. 13( a) illustrates an example implementation of correlating anisosurface topology image dataset of a segmented spine to an acquiredintraoperative surface topology for registering the image datasets, inwhich a transformation matrix is derived.

FIG. 13( b) illustrates an example implementation of combining asurgical plan (block 73) and transforming an image dataset (block 82)for remapping coordinates and updating the surgical plan forimplantation of a surgical interventional device.

FIG. 14 is a flow diagram of an example implementation of a method ofintraoperative surgical feedback guidance, including error checking andcorrective intervention.

FIG. 15 is a flow diagram of an example implementation of a method ofintraoperative surgical feedback guidance including registration using asubset of points of captured image data.

FIG. 16 is a flow diagram of an example implementation of a method ofintraoperative surgical guidance feedback including surface typeidentification and clutter rejection.

FIG. 17 is a flow diagram of an example implementation of a method ofintraoperative surgical guidance feedback including surfaceidentification and tool tracking.

FIG. 18 displays grayscale plots showing typical example results of theiterative registration error process with the convergence of oneregistered optical topology dataset to a subsequent optical topologydataset.

FIG. 19 demonstrates an iterative registration error as it convergencesto the pre-defined confidence criteria of one optical topology datasetto a CT surface registration dataset.

FIG. 20 displays the points, which make up the surface of a spinephantom acquired through optical topology, where these points areuniformly down sampled by spatial position.

FIG. 21 displays the points, which make up the surface of a spinephantom acquired through optical topology, where these points areuniformly down sampled by normal vectors of the corresponding points.

FIG. 22 is an example demonstration of spectral based clutter rejection.

FIG. 23 is an example of color based clutter rejection, showingstructured light images with and without the use of color for therejection of muscular tissue.

FIG. 24 is an example demonstration of surface roughness based clutterrejection, showing structured light reconstruction of bone and muscletissue with and without roughness-based clutter rejection.

FIG. 25 is the integration of tool tracking and surface topology imagingsystem to enable surgical navigation.

FIG. 26 is a schematic of how the coordinates of the differentcomponents of the surgical navigation system are related. The tip of thearrow indicates the components whose position is tracked.

FIG. 27 is an example implementation used in the operating room, wherethe surface topology imaging is a handheld device.

FIG. 28( a) is an illustration of an example system for surface topologydetection and tool tracking using two cameras, where the cameras have adual role of acquiring surface topology and tool tracking data. Thecameras and projectors are rigidly attached to a frame so that a fixedspatial relationship exists between the components.

FIG. 28( b) is an illustration of an example system for surface topologydetection and tool tracking using four cameras, where two of the camerasare used for surface topology imaging, and another two cameras are usedfor tool tracking. The cameras and projectors are rigidly attached to aframe so that a fixed spatial relationship exists between thecomponents.

FIG. 29 is a flow chart illustrating an example method of performingserial measurements of surface topology and tool tracking with anintegrated system.

FIG. 30 is a flow chart illustrating an example method of performing acalibration to relate the coordinate system of the surface topologyimaging system and the tool tracking system.

FIGS. 31( a) and 31(b) show (a) a full surface model of the tool to betracked with center line and tip specified, and (b) marker balls fromtool segmented and centers calculated/specified.

FIGS. 32( a) and 32(b) show (a) a surface topology scan acquired duringa procedure, and (b) the automatic segmentation of marker balls based oncolor.

FIGS. 33( a) and 33(b) provide (a) an illustration of the geometricalrelationships employed to determine the center of a detected ball, and(b) a plot that demonstrates the decrease in the standard deviation ofthe determined marker position with the number of surface normalsemployed in the calculation.

FIG. 34 shows the redisplay of the full tool with center-line and tipspecified after performing landmark registration.

DETAILED DESCRIPTION

Various embodiments and aspects of the disclosure will be described withreference to details discussed below. The following description anddrawings are illustrative of the disclosure and are not to be construedas limiting the disclosure. Numerous specific details are described toprovide a thorough understanding of various embodiments of the presentdisclosure. However, in certain instances, well-known or conventionaldetails are not described in order to provide a concise discussion ofembodiments of the present disclosure.

As used herein, the terms, “comprises” and “comprising” are to beconstrued as being inclusive and open ended, and not exclusive.Specifically, when used in this specification including claims, theterms, “comprises” and “comprising” and variations thereof mean thespecified features, steps or components are included. These terms arenot to be interpreted to exclude the presence of other features, stepsor components.

As used herein, the term “exemplary” means “serving as an example,instance, or illustration,” and should not be construed as preferred oradvantageous over other configurations disclosed herein.

As used herein, the terms “about” and “approximately”, when used inconjunction with ranges of dimensions of particles, compositions ofmixtures or other physical properties or characteristics, are meant tocover slight variations that may exist in the upper and lower limits ofthe ranges of dimensions so as to not exclude embodiments where onaverage most of the dimensions are satisfied but where statisticallydimensions may exist outside this region. It is not the intention toexclude embodiments such as these from the present disclosure.

The following terms used in this description have the followingmeanings:

As used herein, “registration” refers to a process of matching datapoints or sets of data points from various datasets to the samecoordinate system under a set of constraints. Various image datasetsfrom a target image space are aligned to a reference image space. Forexample, a set of points in R³ (three-dimensional space) acquired bysampling an object at different time points and/or using differenttechniques (for example, MRI, CT, positron emission tomography (PET),ultrasound, and back scattered radiation) provide datasets in differentcoordinate systems.

As used herein, “transformation” refers to a process of generating a mapto instruct how to scale, translate, and rotate all points in an objectsuch that the object remains aligned to one of another object and theobject itself, at a different time point and/or imaged with a differenttechnique. A subset of transformations known as “affine transformations”maps points from R³→R³. Such affine transformations can be representedby matrices and are the outputs from the registration.

As used herein, “translation” refers to a shift of a point or set ofpoints in R³ by a fixed distance in the same direction. Translation isone component of transformation.

As used herein, “rotation” refers to a circular motion of a point or setof points around a fixed axis in R³ termed an axis of rotation. Rotationis another component of transformation.

As used herein, “scaling” refers to the enlarging or shrinking thedimension of an object. For uniform scaling, the scale factor is thesame in all directions. Scaling is another component of transformation.

As used herein, “location” refers to the position of a point or anobject in physical space R³ relative to an object (for example, bone,camera, surface structure) in a general area.

As used herein, “orientation” refers to any one of a number of angularpositions relative to a set of reference axes in R³, where one point isheld in a fixed position and around which the object may be rotated.

As used herein, “backscattered radiation” refers to the deflection ofradiation through angles greater than 90° to the initial direction oftravel. Example principles of backscattered radiation for obtainingsurface topology information include, but are not limited to, structuredlight, phase-modulated light, laser triangulation, laser range finding,photogrammetry, and stereoscopy. Backscattered radiation furtherincludes electromagnetic non-ionizing radiation in either the visible orinvisible range (i.e. infrared or ultraviolet).

As used herein, “texture” refers to the characteristics of a surface,which include its representation in color and/or roughness.Specifically, the color texture of a surface is characterized by how itsindividual parts are spectrally perceived by an image capture system,such as a camera. Roughness refers to how individual parts of a surfacebelonging to a predefined region deviate from the mean of that region.

System Overview

Referring now to FIG. 1( a), an example image-based surgical guidancefeedback system 100 is schematically illustrated. System 100 includes: asurface topology backscattered radiation image acquisition system 1, forexample, a structured light illumination, laser range scanning, or lasertriangulation surface topology imaging system; a surgical guidancecontroller 3 in communication with the surface topology imageacquisition system 1; a storage device 2 in communication with thesurgical guidance controller 3, for example, magnetic or solid statemedia, for storing image data and processed data; a display 4, such as acomputer monitor, in communication with the surgical guidance controller3; and a tool tracking subsystem 6, in communication with the surgicalguidance controller 3.

As will be further described below, surgical guidance controller 3registers acquired image data from the surface topology backscatteredradiation image acquisition system 1 to additional, for example,pre-operative, image data from the storage device 2. The registered dataare then provided on the display 4 in an output format including imagedata and additional text-based data such as the registration error, anddistance measurements that indicate the proximity of a surgical tool toa target defined in the surgical plan. In one example, afterco-registration, the backscattered image data may be displayed togetherwith the registered additional image data as a single image. Guidancefeedback can be provided in part through other output user interfacessuch as, for example, speakers or other audible output devices, andlight beams projected directly on the patient showing desired positionof an interventional device to be inserted or attached, such as apedicle screw, or a combination thereof. The system 100 is particularlyadvantageous for surgeries involving orthopedic structures, includingspine, hip, skull, and knee. The system may, for example, be employed toprovide intraoperative guidance for orthopaedic, neurosurgical, head andneck, and otolaryngological surgical procedures.

The forthcoming description describes example implementations of methodsand systems primarily with illustrative reference to applications forguidance feedback in spinal surgery, particularly the insertion ofpedicle screws. The insertion of pedicle screws is used forillustration, because a demanding aspect of pedicle screw insertion isthe identification of the entry to the pedicle canal and thedetermination of the angle of the pedicle canal relative to thesurgically exposed surface of the vertebrae without direct visualizationof the pedicle canal and the vertebrae. Typically, a surgeon exposesonly a portion of the posterior of the vertebral bone through which thepedicle is entered. Failure to enter the pedicle on a proper trajectorycan, for example, result in violation of the walls of the pedicle or theanterior cortex of the vertebrae.

Surgical guidance controller 3 can be, for example, a processing unitand associated memory containing one or more computer programs tocontrol the operation of the system, the processing unit incommunication with a user interface unit 5 and the display 4. In oneexample, surgical guidance controller 3 may be a computing system suchas a personal computer or other computing device, for example in theform of a computer workstation, incorporating a hardware processor andmemory, where computations are performed by the processor in accordancewith computer programs stored in the memory to carry out the methodsdescribed herein. For example, the processor can be a central processingunit or a combination of a central processing unit and a graphicalprocessing unit.

Surgical guidance controller 3 records and processes backscatteredradiation from the surface topology of the rigid surgical structure ofinterest and, utilizing the preoperative image inputs above, operates,for example, to provide real-space spatial relationships of the surgicaltarget to the preoperative 3D image dataset and an optional surgicalplan that reflects current intraoperative geometry. Example methods ofprocessing acquired surface topology data to register the surfacetopology data to pre-operative 3D image data are described in furtherdetail below. Surgical guidance controller 3 may also optionallydetermine the real-space spatial relationship of a surgical tool inrelation to the intraoperative geometry of the target rigid surgicalstructure of interest, as described in more detail below.

In one embodiment, system 100 includes a general purpose computer or anyother hardware equivalents. Thus, the system may include at least oneprocessor (CPU/microprocessor), a memory, which may include randomaccess memory (RAM), one or more storage devices (e.g., a tape drive, afloppy drive, a hard disk drive or a compact disk drive), and/or readonly memory (ROM), and various input/output devices (e.g., a receiver, atransmitter, a speaker, a display, an imaging sensor, such as those usedin a digital still camera or digital video camera, a clock, an outputport, a user input device, such as a keyboard, a keypad, a mouse, aposition tracked stylus, a position tracked probe, a foot switch,6-degree input device based on the position tracking of a handhelddevice, and the like, and/or a microphone for capturing speech commands,etc.). In one embodiment, surgical guidance controller 3 is implementedas a set of instructions which when executed in the processor causes thesystem to perform one or more methods described in the disclosure.

Surgical guidance controller 3 may also be implemented as one or morephysical devices that are coupled to the CPU through a communicationchannel. For example, surgical guidance controller 3 can be implementedusing application specific integrated circuits (ASIC). Alternatively,surgical guidance controller 3 can be implemented as a combination ofhardware and software, where the software is loaded into the processorfrom the memory or over a network connection. In one embodiment,surgical guidance controller 3 (including associated data structures) ofthe present disclosure can be stored on a computer readable medium,e.g., RAM memory, magnetic or optical drive or diskette and the like.

While some embodiments have been described in the context of fullyfunctioning computers and computer systems, those skilled in the artwill appreciate that various embodiments are capable of beingdistributed as a program product in a variety of forms and are capableof being applied regardless of the particular type of machine orcomputer-readable media used to actually effect the distribution.

Examples of computer-readable media include but are not limited torecordable and non-recordable type media such as volatile andnon-volatile memory devices, read-only memory (ROM), random accessmemory (RAM), flash memory devices, floppy and other removable disks,magnetic disk storage media, optical storage media (e.g., Compact DiskRead-Only Memory (CD ROMS), Digital Versatile Disks, (DVDs), etc.),among others. The instructions can be embodied in digital and analogcommunication links for electrical, optical, acoustical or other formsof propagated signals, such as carrier waves, infrared signals, digitalsignals, etc.

A machine-readable medium can be used to store software and data whichwhen executed by a data processing system causes the system to performvarious methods. The executable software and data can be stored invarious places including for example ROM, volatile RAM, non-volatilememory and/or cache. Portions of this software and/or data can be storedin any one of these storage devices. In general, a machine-readablemedium includes any mechanism that provides (i.e., stores and/ortransmits) information in a form accessible by a machine (e.g., acomputer, network device, personal digital assistant, manufacturingtool, any device with a set of one or more processors, etc.).

Some aspects of the present disclosure can be embodied, at least inpart, in software. That is, the techniques can be carried out in acomputer system or other data processing system in response to itsprocessor, such as a microprocessor, executing sequences of instructionscontained in a memory, such as ROM, volatile RAM, non-volatile memory,cache, magnetic and optical disks, or a remote storage device. Further,the instructions can be downloaded into a computing device over a datanetwork in a form of compiled and linked version.

Alternatively, the logic to perform the processes as discussed abovecould be implemented in additional computer and/or machine-readablemedia, such as discrete hardware components including large-scaleintegrated circuits (LSIs), application-specific integrated circuits(ASICs), or firmware such as electrically erasable programmableread-only memory (EEPROMs).

The controller can further include a clutter identification module toidentify clutter in the acquired backscattered image data.

The controller can further include a confidence criteria module todetermine if registration is occurring within a pre-set confidencecriteria, and if not, intervention may be sought to provide additionaldata to be used in intraoperatively registering.

Referring to FIG. 1( b), an example implementation of an image-guidedspinal surgical procedure using the image-based surgical guidancefeedback system 100 is provided. System 100 may include, for example, auser workstation 7, incorporating surgical guidance controller 3 and thememory storage device 2 to carry out the methods described herein. Userworkstation 7 may consist of display 4, such as a high definitionmonitor, the surgical guidance controller 3, and user interface 5, suchas a keyboard, for inputting instructions into the system 100. Allcomponents can be installed into a single unit, such as a medical gradecart 11. In this implementation, the system further comprises twocameras 12 for detecting the structured light grid pattern 13 emittedfrom the digital projector 15, which is incident on the subject 16. FIG.6( a) illustrates a specific example in which a portion of an exposedspine is imaged using a structured light pattern to determine and recordthe three-dimensional surface profile for co-registration withpre-operative 3D image data.

A tool 17 (for example, a surgical tool, probe, surgical instrument, orother freely movable item), having fiducial markers 18 adhered thereto,may also be integrated with system 100, in order to co-register theposition of tool 17 with 3D pre-operative image data. As furtherdescribed below, the position and orientation of tool 17 may bedetermined using an additional global position sensor, or alternativelymay be determined using measurements obtained from surface topologybackscattered radiation image acquisition system 1.

Referring now to FIG. 1( b), an example implementation of animage-guided spinal surgical procedure using the image-based surgicalguidance feedback system 100 is provided. The image-based surgicalguidance system 100 can be implemented using, for example, abackscattered radiation surface topology imaging device, at least oneregistration algorithm, and a software module to provide intraoperativehigh-speed feedback information to the clinician for planning andexecuting the surgical procedure. Optionally, color textures of objectswithin a field of view can be captured either simultaneously by thebackscattered radiation imaging device, or separately by a secondimaging device, for improving the accuracy and speed of theregistration. The surface topology information can be registered to the3D preoperative imaging dataset to provide structural information aboutthe surgical structure of interest that is not visible to the clinician.While system 100 can be used with a set of fiducial markers placed onthe structure of interest for tracking during imaging, it is anadvantage of the present system that fiducial markers are not requiredfor surgical guidance.

System 100 may include, for example, a user workstation 7, incorporatingthe surgical guidance controller 3 and the memory storage device 2 tocarry out the methods described herein. User workstation 7 may consistof display 4, such as a high definition monitor, the surgical guidancecontroller 3, and user interface 5, such as a keyboard, for inputtinginstructions into the system 100. All components can be installed into asingle unit, such as a medical grade cart 11. In this exampleimplementation, the system further comprises two cameras 12 fordetecting the structured light grid pattern 13 emitted from the digitalprojector 15, which is incident on the subject 16.

As shown in FIG. 1( b), the surface topology backscattered radiationimage acquisition system 1 may include at least one camera 12, andpreferably two cameras 12. While the system 1 is operable with a singlecamera 12, the inclusion of two cameras 12 can increase the field ofview and surface coverage (with fewer blind spots). Using multiplecameras can also enable spectroscopic imaging via the inclusion offilters. Imaging frame 20 may be optionally provided to house bothtopology imaging system 1 (for example, a digital projector 15 where astructured light source is used) and the one or more cameras 12. Thesurface topology information acquired by this system is registered tothe 3D preoperative imaging dataset to provide information relating tosub-surface structure and composition that would be otherwise hiddenfrom the operator's view.

Backscattered Radiation Topology Systems

According to example methods provided herein, backscattered radiationtopology and texture-based surgical tracking and navigation can beenabled during surgery, for example, for feedback guidance in theplacement of the surgical interventional device to the surgicalstructure of interest (e.g. attachment of fixation devices to the spineduring spinal surgery). Backscattered radiation, includingelectromagnetic non-ionizing radiation in either the visible orinvisible range, can be utilized for acquisition of surface topologyimage data. The use of light outside the visible range (i.e. infrared orultraviolet) may be beneficial so the field of view of a surgeon is notdistracted. Appropriate safety precautions may be required when usinginvisible light. Using a 3D backscattered imaging device, topology mapsof real-space surgical surfaces can be created for exposed regions ofinterest. Correspondingly, by image registration, structural informationbeneath the surface that is hidden from the surgeon's view is provided.

The 3D backscattered radiation surface topology imaging devices mayemploy, but are not limited to, structured light illumination, lasertriangulation, laser range finding, time-of-flight/phase modulation,photogrammetry, and stereoscopic methods. Some further example detailsof such example methods include:

i) Photogrammetric devices: Multiple images of a region of interest areacquired from different angles by either moving a single camera or byusing multiple fixed cameras. Based on acquired images, a surfacetopography map of a region can be generated.

ii) Laser triangulation devices: A collimated laser beam can be used toilluminate a point on a target surface of interest. A portion of thelight is reflected from the surface and is detected using for example, acharge-coupled device (CCD) camera. The position of the detected lighton the CCD camera can be used to determine the distance to that point onthe object. Scanning the laser beam across the surface using a devicesuch as a galvo scanner will yield a surface topography map.

iii) Time of flight/phase modulation devices: A laser can be passedthrough a phase modulation device and then split into two arms of aninterferometer. One arm of the interferometer contains a mirror(reference arm) and the other can be sent to the object surface beingscanned (sample arm). Sample arm optics collimate/focus the laser to apoint on the surface and galvo scanners can be used to scan the regionof interest. The reflected light from the sample and reference arm beamscan be recombined and measured by a photodetector. The relativedisplacement of target in sample arm to the mirror in reference armleads to a phase shift in the measured signal relative to the modulationfrequency of the phase modulator. This phase shift map can then bedirectly converted to a surface topography map.

iv) Structured light photography devices: A region of interest isilluminated with one or multiple patterns either by using a fringe(sinusoidal) pattern or binary pattern generated, for example, byinterference of two laser beams or using a digital projector. Using oneor multiple camera(s), images can be acquired to record the projectedpattern that appears on the object. Using knowledge of how the patternappears on a flat surface and known orientations of the camera(s) andprojector, deformations in the pattern allows a surface topography mapof the target to be generated, as further described below. Such deviceswill be known to those skilled in the art, and are described in Salvi(J. Salvi, “Pattern Codification Strategies in Structured LightSystems”, Pattern Recognition (37), pg. 827-849, April 2004) and Zhang(S. Zhang, “High-Resolution, Real-Time Three-Dimensional ShapeMeasurement”, Optical Engineering 45(12), 123601, December 2006).

Colour Filters for Spectral Processing

Optionally, color textures of objects within the field of view, capturedeither simultaneously by the topology system 1 or separately by anothercamera, can serve as additional features that can be extracted toimprove accuracy and speed of registration.

Referring to FIG. 2, to improve surface identification, a filter 40 canbe integrated into the cameras 12 to preferentially accept only certainbands of the electromagnetic spectrum. The filters 40 can be optimizedto achieve maximum contrast between different materials and thus improvethe clutter identification process, as further described below. Forexample, bands that are common to backscattered radiation from typicalclutter items, the surgical structure of interest, and the surgicaltool(s) can be filtered out such that backscattered radiation of highcontrast between clutter items, surgical structure and surgical toolscan be acquired.

A filter 40 may be fixed in front of a given camera 12, or may bemovable. For example, a filter 40 may be slidably movable into and outof the optical path of camera 12, manually or in an automated fashion(such as driven by a motor or a solenoid). In another example, multiplefilters may be periodically positioned in front of a given camera inorder to acquire spectrally resolved images with different spectralranges at different instants in time, thereby providing time dependentspectral multiplexing. Such an embodiment may be achieved, for example,by positioning a plurality of filters in a filter wheel that iscontrollably rotated to bring each filter in the filter wheel into theoptical path of the camera at different moments in time.

3D Image Dataset

Image dataset provided to system 100 can include any of the followingnon-limiting examples: preoperative 3D image data of a surgicalstructure of interest, such as the spine, in a subject acquired, forexample, using any one of PET, CT, MRI, or ultrasound imagingtechniques; a preoperative surgical plan developed by a clinicalpractitioner (for example, a surgeon), and a surface topology imagedataset, optionally including texture data, of the rigid surgicalstructure of interest.

In an example implementation, intraoperative CT imaging is used toacquire the preoperative image dataset. FIG. 3( a) illustrates exemplaryCT image slices of the torso. Imaging modalities such as MRI,ultrasound, and other 3D imaging methods are also applicable foracquisition of preoperative image datasets. These image datasets can beused to develop a surgical plan for implantation of the surgicalinterventional device (e.g. a spinal cage or pedicle screw) into adesired position in the surgical structure of interest, and also serveas a reference dataset of a subject's anatomy.

The image data for the surgical structure of interest, for example, thespine, may be segmented before surgery and reconstructed as an imagedataset for a rigid surgical structure. In several non-limitingexamples, the structure can be a bone structure, such as a spinalcolumn, a skull, a hip bone, a foot bone, and a patella. For example,FIG. 3( b) is a schematic of a posterior orientation of a segmentedspine; FIG. 3( c) is a schematic of a lateral orientation of thesegmented spine; and FIG. 3( d) is a schematic of a cross-sectionalorientation of the segmented spine. The segmented surgical structure ofinterest image data serve as a template for registration withbackscattered radiation topology data acquired intraoperatively. Examplemethods for segmentation are described in further detail below.

Incorporation of Surgical Plan

An example implementation of surgical guidance through the use ofbackscattered radiation topology imaging can include, for example,acquiring preoperative imaging data and developing a surgical plan,performing intraoperative imaging, and, in combination with thepreoperative image data, generating useful information to guide surgeryin the form of co-registered images, and displaying or otherwisecommunicating this surgical guidance information to a surgeon oroperator. A preoperative plan can be developed, for example, by aclinician using the preoperative image data, and made available for usein the system. This example implementation enables repetition ofintraoperative imaging and generating guidance feedback.

A preoperative surgical plan may consist of, for example, the desiredposition and orientation of pedicle screws defined with respect to apreoperative image data (e.g. CT, MRI) of the patient. The plan would bedeveloped by a surgeon before a surgery, by analyzing the geometry ofthe vertebrae of interest, and selecting screws with the correctdimensions (e.g. length and radius) in accordance with the volume of thepedicle. The choice of screws and their positions would take intoconsideration the surrounding tissues to avoid damaging critical nervesand blood vessels or to make sure the screw does not breach thevertebral wall and enter the surrounding tissue. During surgery, thepreoperative plan is updated to reflect the intraoperative geometry of apatient's spine with the optimal trajectory and a cone of acceptance,described below, as a guide to assist a surgeon during pedicle screwinsertion.

System 100 may provide, through a display and/or user interface,guidance to assist in the placement of a surgical interventional deviceby providing intraoperative image feedback of changes in an orientationof a surgical structure of interest during the surgical procedure. Byway of example and referring to FIG. 4, one example parameter, a cone ofacceptance 25, can be used to improve accuracy of implantation of apedicle screw into a vertebrae 23. The cone of acceptance 25 is definedby a range of trajectories relative to the vertebrae 23, along which thepedicle screw can be securely implanted into the pedicle canal withoutdamaging the spinal cord 24, sparing the surrounding peripheral nervesand blood vessels, and does not protrude out of the bone. The range ofavailable trajectories has limited lateral and angular freedom in lightof the narrow middle section of the pedicle canal. Taken together, thetrajectories collectively define a frustum conical shape with a widerend at an entry surface of the vertebral arch.

The range 28 of available trajectories relative to a vertebra 23 isdependent on: (1) the dimensions of the vertebra; (2) the orientation ofthe vertebra 23; and (3) the size of the pedicle screw. The cone ofacceptance 25 incorporates the narrowest section of the pedicle canal,along a principle axis 26, for defining the optimal surgicalimplantation site for safe and secure pedicle screw insertion.

The cone of acceptance 25 is typically determined as part of apreoperative surgical plan. Methods for incorporating the surgical planinto the intraoperative guidance system are addressed below. The system100 monitors the orientation of the vertebra 23, which changes duringsurgery by a number of means, such as during drilling of the vertebraand depression of the spine by the surgeons, and generates guidancefeedback, such as an update and display of the cone of acceptance 25,providing an example of motion correction. This example display ofguidance feedback with motion correction is illustrated in FIG. 4. Thecenter of the cone of acceptance is represented by a single trajectoryreferred to as the principle axis 26.

FIGS. 5, 6(a) and 6(b) illustrate a schematic of acquisition of atopology map of the exposed spine 19 through illumination with astructured light grid pattern 13, via the digital projector 15 and twocameras 12. Imaging frame 20 houses digital projector 15 and the cameras12. The use of a backscattered radiation topology system (in thisexample, using structured light) enables the dynamic tracking of thesurface of interest, and optionally, the dynamic updating of a surgicalplan, without requiring the use of a physical coordinate frame orfiducial markers being rigidly attached to the surface. It is to beunderstood that the structured light system need not include twocameras, and may be provided with a single camera.

The non-contact, fiducial-free detection of the surface of interestenables the dynamic tracking of the surface in an intraoperativeenvironment. For example, referring to FIG. 6( b), the position of thevertebra 23 can shift to a new position 23′ relative to thepreoperatively determined position due to effects from surgicalintervention (for example, pressure applied to the vertebra 23 as afinger of hand 29 is applied to the surface; such pressure could beprovided or released by many other examples, such as a drill, notshown). An updated position of the vertebra 23′ can be determined andoutputted by the system 100 on the display 4. There is a concurrentshift in the principle axis 26 to an updated principle axis position 26′and also in the position of the spine 19 to an updated spinal columnposition 19′.

Referring now to FIG. 7, an example shift in the position of thevertebra 23 from a preoperative position 23 to an intraoperativeposition 23′ is illustrated. The vertebrae 23 preoperative position isused to develop the surgical plan. In developing the surgical plan, theprinciple axis 26 is determined to ensure avoidance of the spinal cord24. The preoperative positions of the structures are indicated withsolid lines. During surgery, positions can shift due to, for example,surgical intervention and change in subject position, as noted above.The updated locations of the target vertebrae 23′, principle axis 26′,and spinal cord 24′ are determined by the system 100 and outputted onthe display 4. Accordingly, system 100 provides a dynamically updatedsurgical plan that is registered to the patient anatomy in real-time.

Intraoperative image updates of the vertebrae 23 can be providedcontinuously or discretely according to input into the system 100 by,for example, a surgeon. In the situation where updates are providedcontinuously, the system 100 can operate autonomously obviating the needfor the surgeon to input any additional data. In the situation whereupdates are provided discretely, for example updates provided at singletime points, the surgeon can request an image data update by inputting arequest into the system 100. The updated plan is provided on the displaydevice 4 on command without any other user interface. The updated imagedata and related updated intraoperative surgical plan enable a surgeonto accurately implant, for example, a pedicle screw into a vertebra 23.

In one example, the surgical plan may include surgical criteria that canbe displayed on the co-registered image. Examples of criteria that thesurgeon may input into system 100, as part of a surgical plan, include,but are not limited to: the accepted accuracy of screw placement; thecoordinates of the point of entry into the vertebra 23 that define theprinciple axis 26; the accepted angle of screw placement; and the depthof screw placement.

Referring to FIG. 4, these criteria can be used to calculate a plane ofsmallest diameter (for example the narrowest section of the pediclecanal), through which the principle axis 26 runs centrally. Due to thespatial registration between the surface of interest and the projector,the calculated plane 27 can then be projected onto the surface of thevertebrae via the projector to provide a desired solution for pediclescrew placement 28. The cone of acceptance 25 coordinates can then beoverlaid onto the vertebrae and provided on the display 4. The system100 can remain in “standby mode” until the structure of interest issurgically exposed.

Surface Detection and Image Registration for Intraoperative Guidance

Referring now to FIG. 8, an example method performing image registrationand guidance using a backscattered radiation surface guidance system isillustrated. As described above, optical topology imaging and surfacetopology image data processing algorithms are employed to track thelocation and orientation of a rigid structure of interest during asurgical procedure. While the examples below relate to orthopaedicsurgical procedures, it is to be understood that the methods may beapplied to a wide range of surgical procedures and other applications.

As shown at step 50 in FIG. 8, the method initially involves obtainingpreoperative image data acquired by any one of a number of imagingmodalities, and optionally developing a preoperative surgical plan.Intraoperative topology imaging data is then acquired in step 51 andregistered to the pre-operative image data for providing guidancefeedback to guide the surgical procedure intraoperatively. In step 52, asurgical plan may be updated based on a shift in the position of thestructure of interest as detected by system 100. Steps 51 and 52 arerepeated as necessary during a surgical procedure. This method isdescribed in further detail below, with reference to FIGS. 9 to 17.

(i) Preoperative Image Acquisition and Surgical Planning Module

Referring to block 50 of FIG. 9, a 3D image dataset is acquiredpreoperatively to locate an anatomical region of interest by any one ofa number of 3D imaging modalities, including, but not limited to, MRI,CT, and ultrasound. Certain imaging modalities may be more suitable fora given surgical context depending on the primary target of interest.For example, CT imaging is suitable when the primary target of interestis the spine. The spine is segmented from the 3D images forintraoperative image registration. The individual vertebrae aresegmented, which can include labeling, either automatically or manually,with the correct anatomical location. Individual vertebrae parts (e.g.the laminar or pedicle) may be further segmented for implantation device(e.g. pedicle screw) placement planning. These steps of segmenting theindividual vertebra are further described below.

In step 53 of the example method shown in FIG. 9, the preoperative imagedata of the orthopaedic structures of interest is acquired (for example,a CT scan of the patient's spine). The CT image dataset is processed togenerate image data of one or more surfaces, such as an isosurface. Theprocessing results in preoperative image data that can be, for example,a polygonal mesh output data of the spine (sometimes referred to hereinas CT_MESH_FULL). Isosurface generation can, for example, use apredefined threshold parameter distinguishing differential based tissuedensity, such as bone, compared to soft tissue density.

A preoperative plan is then developed in step 55 and made available tosystem 100, using the preoperative surface image data (CT_MESH_FULL) todetermine and record the planned location and orientation of a surgicalintervention device. The preoperative plan can, for example, alsospecify acceptable error associated with each intervention.

The preoperative image data for the orthopaedic structure is then, instep 54, segmented (manually or automatically) into structure segmentsthat have rotational and translational degrees of freedom with respectto one another (e.g. individual vertebrae). Segmentation can beperformed manually, for example, by a radiologist, or with the aid ofsemi-automatic tools. Regardless of the segmentation method, the outputof this step provides a given number, N, of preoperative surface imagedata segments (sometimes referred to herein as CT_MESH_OB_(—)1 . . .CT_MESH_OB_N). The segmented image data are made available to thesystem.

The segmented preoperative image data can then be registered to thepreoperative plan in step 56. Each segment is registered to theassociated aspect of the plan. It is to be recognized that multipleaspects of plans could be registered against one segment (for example,two pedicle screw holes in a vertebrae) and still further aspects ofplans could be registered against multiple segments (two pedicle screwholes registered on one segment, and two screw holes on anothersegment). Surgical guidance feedback could be provided on the variousaspects independently as needed, or simultaneously. For example, thesurgical intervention for the plan can include an entire device andattachment points. As a further example, the surgical intervention caninclude planned attachment points or principle axis only, such as one ormore drill holes or cut lines.

The combined orthopaedic and preoperative plan data, as described above,thus includes the segmented preoperative image of an orthopaedicstructure and a preoperative plan allowing this combined representationto be rotated and translated as desired.

(ii) Topology Data Acquisition and Dataset Manipulation Module

Backscattered radiation surface topology data of the exposed structureis obtained in step 57 of FIG. 9. The topology data can be capturedcontinuously or on demand. Each of the preoperative image dataorthopaedic segments can be registered to the backscattered radiationtopology scan in step 58.

One particular example method for the registration, as shown in the flowchart provided in FIG. 10, is based on iterative closest point (ICP)registration, which is one of the most commonly used surfaceregistration techniques. ICP registration technique requires two inputs:backscattered radiation topology data 200 and structure segment imagedata 201. Dependent on the imaging scenario, these data can initially gothrough an optional processing 202 step to remove clutter or to identifyspecific components. This clutter removal step is described furtherbelow. Other suitable methods of surface registration are described inChen and Medioni (Y. Chen and G. Medioni, “Object Modeling byRegistration of Multiple Range Images”, Proc. IEEE Conf. on Robotics andAutomation, 1991) and Besl and McKay (P. Besl and N. McKay, “A Methodfor Registration of 3D Shapes”, IEEE Trans. Pattern Analysis and MachineIntelligence 14 (1992), 239).

ICP is suitable for relatively small transformations when using complexpoint clouds. Thus, a coarse estimate of the transformation must be madeinitially on the first pass through the algorithm {i==1} 203. For thispurpose an interactive point-based approach can, for example, be used toderive the initial transformation T_initial in steps 204 and 205. Forexample, M (where M>=3) representative points are selected from each ofthe segmented isosurface image datasets (CT_MESH_OB_(—)1 . . .CT_MESH_OB_N), where these matched points serve as virtual fiducialmarkers in both the CT isosurface and backscattered radiation surfacetopology datasets 204. Using a singular value decomposition algorithm205, the M points can be brought into alignment providing an initialtransformation for a more precise registration using the high-resolutiondataset, for example, as described in Salvi (J. Salvi, “A Review ofRecent Range Image Registration Methods with Accuracy Evaluation”, Imageand Vision Computing 25 (2007) 578-596). Alternatively, this initialalignment can be performed using the method described in Berthold K. P.Horn (1987), “Closed-form solution of absolute orientation using unitquaternions”. Next, each of the vertebrae 23 meshes (CT_MESH_OB_(—)1 . .. CT_MESH_OB_N) is registered to the backscattered radiation datasetsusing ICP in parallel with T_initial as an initial condition for thetransformation 206.

Although T_initial, the initial transformation, can be derived for eachiteration, a possible implementation includes obtaining T_initial onlyonce. For each new backscattered radiation dataset {i>1} 203 that isacquired, the last transformation matrix {i−1} calculated 207 for thevertebrae 23 of interest can be used as the starting point for the newregistration 208, rather than the original registration as the startingpoint, saving memory, computation time, and reducing overall algorithmcomplexity.

An example implementation to improve processing speed duringintraoperative guidance involves utilizing a subset of points from theimage data instead of the complete point cloud (complete image dataset)for image registration, as shown in FIGS. 14 and 15. These points may beselected automatically by analyzing the topology map for unique featuresthat best represent a target structure of the body. For instance,surfaces greater than a predefined curvature may be used (i.e. pointedsurfaces).

The method of sub-sampling is further described as follows. Let P={p₁,p₂, . . . , p_(m)} and Q={q₁, q₂, . . . , q_(n)} be the two surfaces tobe registered, with m and n points respectively. For this example, thesurface P will be aligned towards Q. Finding matching points betweenthese 2 surfaces requires that for each point in P, a correspondingclosest point is located in Q. In the simplest case, all the points in Pare matched to a point in Q. However, due to the density of pointsavailable in the surfaces, practically, only a subset of the points in Pis needed for point matching without significantly affecting theaccuracy of the registration. The main advantage of sub-sampling thedata is a decrease in the time required for computing the registration.Furthermore, it can also act as a step to select relevant features fromthe surfaces, as further described below, and as described byRusinkiewicz and Levoy (Efficient Variants of the ICP Algorithm (3DIM2001):145-152, Quebec City, Canada, (May 2001)).

According to one example method, the points on a given surface may beselected randomly until a minimum number of points are reached.

Following another example, given the three-dimensional (3D) position ofthe points (the x, y, z coordinates), the subset of points may beselected so that they are uniformly sampled in space. FIG. 20demonstrates an example of this uniform down sampling by spatialposition, where the percentage (100%, 33%, 20%, 10%) 320, represents theremaining points post-down sampling. In the Figure, the points whichmake up the surface of a phantom spine acquired through optical topologyare down sampled uniformly by spatial position.

In a third example method, each point in the surface has a correspondingnormal. The normal is a vector that is perpendicular to the tangentplane of the surface at that point. Instead of using the spatiallocation (as in the preceding example), sampling can be performed basedon the distribution of the normal vectors. FIG. 21 shows an example ofthis uniform down sampling using normal vectors of corresponding points,where the percentage (100%, 33%, 20%, 10%) 330, represents the remainingpoints post down sampling. As demonstrated in FIG. 21, the surface of aphantom spine, acquired through optical topology, is down sampleduniformly by normal vectors of the corresponding points. In this casewhen the surface topology is relatively slowly varying (i.e. smooth),this method can assign more points to prominent surface features.Therefore, it can improve the accuracy of registering surfaces that aremostly smooth with sparse features.

The output from the registration process can be a transformation matrixincluding translation and rotation identities, such as, for exampleroll, pitch and yaw, for each of the segmented structures. For example,translation identities can be present on an x, y, z coordinate systemwith rotation represented by roll, pitch, and yaw identities. It isrecognized that different transformation matrices can be based onalternative coordinate systems.

The transformation matrices derived can be applied to the combinedsegmented orthopaedic structures and corresponding registeredpreoperative plan to update and to match the orthopaedic structures tothe preoperative plan. This updated structure and plan can then beoutput in the form of images, with optional text, for example,descriptive text relating to relative distances and/or orientations. Theoutput can be on a hardware display, such as a monitor or a head mounteddisplay. Images can be displayed, for example, as slices in twodimensions or in a three-dimensional perspective view to provideguidance feedback. Such surgical guidance feedback can be used, forexample, by a surgeon intraoperatively to assist in guiding anorthopaedic procedure. An example includes the presentation of motioncorrection to the surgeon as a pedicle screw is inserted into a vertebra(as shown in FIG. 7).

Example Implementation of Guidance System for Spinal Surgical Procedure

An example implementation of the surgical guidance system 100, includingimage registration, will now be described. This operational descriptionis based on a structured light example for implantation of a pediclescrew in a vertebra of the spine. If other techniques are used, theoperation can vary as individual components of surface topologyacquisition of backscattered radiation are different. For instance, iftopology is acquired via a laser range device, the projector 15 can bereplaced by a laser range finder. This operational description is by noway limiting, and serves as an example as to how an example guidancesystem can operate.

Prior to surgery, the preoperative image dataset of the spine isacquired via an imaging modality such as CT. The surgical plan isdeveloped by the surgeon based on the preoperative image data, which areinputted into the operator workstation 7 via the user interface 5. Asample preoperative image dataset 72 is illustrated in FIG. 12. Thesedata are segmented and labeled preoperatively 70.

The surgical guidance controller 3 receives and inputs backscatteredradiation image datasets acquired via the digital projector 15. A samplesurface topology image dataset 71 is illustrated in FIG. 12. As will bedescribed below, the preoperative image dataset is processed by thesurgical guidance controller 3 to provide, for example, real-spacespatial relationships between the surgical structure of interest and thepreoperative 3D image dataset and preoperatively developed surgical plan73 to provide current (e.g. real time) intraoperative data with respectto the vertebrae 23.

The surgical plan data may be inputted manually by a surgeon or otheroperator and uploaded by the system 100. Data regarding the segmentationof the structure of interest (e.g. a portion of the spine 19) can bemanipulated by the system 100 upon user instruction via the userinterface 5, such as a keyboard, and subsequently processed by thesurgical guidance controller 3. Such segmentation methods are known tothose skilled in the art. The system 100 can process this data to outputposterior, lateral, and cross-sectional views of the spine region to theoperator on the display 4. Alternatively, the surgical plan can bedeveloped by a surgeon using a computing device, such as a personalcomputer, in advance of the surgical procedure, and the surgical plancan be uploaded to the user workstation 7 prior to surgery. The userworkstation 7 is provided in the operating room during the surgicalprocedure.

The system 100 may then be employed to segment the spine 19 to focus onthe target vertebra 23 into which the pedicle screw will be implanted.Example methods for performing this step are provided below. In thepresent non-limiting example in which the surgical plan involves theplacement of a pedicle screw, the preoperative surgical plan is providedin order to identify an entry point of the pedicle screw into thevertebra 23. The physical dimensions of the pedicle screw (i.e. size,thread count, etc.) are taken into consideration by the surgeon to avoidsurrounding organs and tissues (i.e. spinal column 24 and/or bone exit).The calculated surgical coordinates may be inputted into the systemmanually by the user via the user interface 10 and then processed by thesurgical guidance controller 9 to output the “cone of acceptance” 25coordinates.

Once the vertebrae are surgically exposed and the field of view clear,the process of surface topology image acquisition of the vertebrae isinitiated, for example, by user input. Referring to FIG. 1( b),projector 15 emits light onto the exposed spine 19 in the form of thestructured light grid pattern 13. The cameras 12 acquire the surfacetopology image as part of surface topology system 1. The preoperativeplan and surface topology image dataset for the vertebrae along with theplanned pedicle screw orientation are then displayed on display 4, asshown in FIG. 6( a). The surgeon can then proceed to begin the surgicalprocedure based on the information displayed.

System 100 may acquire surface topology information pre-operatively aswell as intraoperatively. FIG. 12 illustrates a combined surgical plan73 that integrates the preoperative image data 72 with theintraoperative surface topology data 71. In performing the surgicalprocedure, a surgeon can apply pressure to the vertebrae, causing thepreoperatively determined position of the vertebrae to be physicallydisplaced to a new position. Other external factors that can cause ashift in the vertebrae during the surgical procedure include movement ofthe subject or change in position of the spine relative to the positiondetermined from the preoperative CT scan data. The shift in position ofthe vertebrae results in a physical displacement of the principle axis26, which is used to guide pedicle screw placement during the surgicalprocedure. There is an inherent potential for error in the position ofpedicle due to such a shift, and the error can result in pedicle screwexit from the bone or infiltration of the pedicle screw into the spinalcord. The registration and transformation methods disclosed hereinsupport updating intraoperative image data and, optionally, a surgicalplan, to compensate for the induced displacement, as shown in FIG. 6(b).

Referring now to FIG. 9, once a preoperative imaging dataset is acquiredin block 53, the orthopaedic structures (e.g. individual vertebrae) aresegmented in block 54 from the preoperatively acquired CT image dataset.Suitable segmentation algorithms are known to those skilled in the art.For example, a suitable segmentation algorithm is as described by YiebinKim and Dongsung Kim (Computerized Medical Imaging and Graphics33(5):343-352 (2009)). The segmentation algorithm may extractinformation about the vertebrae 23 from the spine in four mainprocessing modules: (1) pre-processing, (2) inter-vertebral disc search,(3) 3D fence generation, and (4) fence-limited labeling, as furtherdescribed below.

In the first processing module, namely the pre-processing module,features such as 3D valleys are detected and valley-emphasized Gaussianimages are outputted. Gradients may be used, however, valleys canprovide better features than gradients in separating two closelyseparated objects because they appear in the middle of two adjacentobjects, while the gradients are detected at the boundaries of everyobject. The steps involved in the pre-processing module include: i)detection of a 3D morphological valley, ii) generation of a valleyemphasized dataset, iii) generation of an intensity based threshold, andiv) generation of x a 3D Gaussian filter, as described in Kim and Kim(Kim and Kim, (2009) Computerized Medical Imaging and Graphics33(5):343-352).

An inter-vertebral disc search module automatically may be employed toextract the spinal cord and detect inter-vertebral discs along thecenter line of the spinal cord. The steps of the inter-vertebral discsearch module include: i) extraction of the spinal cord using aniterative sphere growing algorithm; and ii) extraction of theintervertebral discs by determining the center of each sphere, whichconsists of the extracted spinal cord defining the center line C(t) ofthe cord, and the intensity profile in the plane, which is normal to thecenter line, as described by Kim and Kim (Kim and Kim (2009)Computerized Medical Imaging and Graphics 33(5):343-352).

The 3D fence generation module generates boundary surfaces, used toseparate one vertebra from another. In generating the 3D fence, anerroneous curve that is derived from a local minimum in the optimizationprocess is detected with an evaluation method and then corrected by aminimum cost path finding method, which can then find the globalminimum. The steps in 3D fence generation include: i) generation of a 2Dintervertebral segmentation curve; ii) propagation of the 2D curve intoa 3D surface; and iii) detection and correction of any erroneouslypropagated curves, as described by Kim and Kim (Kim and Kim (2009)Computerized Medical Imaging and Graphics 33(5):343-352). The results ofautomated segmentation can be reviewed by a surgeon prior to beinginputted into the system.

In another example method, a user may perform segmentation of relevantvertebrae manually. FIG. 11 illustrates a sample manual segmentation ofa portion of the spine. Manual segmentation is achieved by the userdrawing fences 68 manually to separate each vertebra 23 at the adjacentvertebral disc(s) 67. This manual separation can occur at the immediatevertebral disc 67 or multiple vertebral discs away, dependent on therequirements of surgery. A fence limited seed region growing module canbe used to extract the vertebrae 23 from the fence bounded region.

A fence-limited labeling module can be employed to label each vertebralvolume using a fence-limited seed region growing (SRG) method. Thevolume is repeatedly expanded from a seed point until a growing pointreaches a 3D fence and its gray value is within homogeneous volumethresholds, as described by Kim and Kim (Kim and Kim (2009) ComputerizedMedical Imaging and Graphics 33(5):343-352). Use of this module is basedon whether the starting planes were inputted manually or areautomatically generated.

The preoperative surgical plan and segmented preoperative imagingdataset can then be combined. FIG. 11 illustrates a sample outputupdated surgical plan using the system 100. The illustrated segmentedand labeled preoperative image dataset 70 includes five vertebrae 23that have been segmented from the preoperative CT dataset and labeledA-E, and also corresponding inter-vertebral discs 75. Followingsegmentation, an isosurface image of the segmented spine 71 can begenerated.

To extract an isosurface dataset for each vertebra 23 from thecorresponding segmented vertebrae dataset, a user can specify a contrastlevel for the vertebrae by entering the information into the system. Thecontrast level would typically lie between 1100-1200 Hounsfield units. Amarching cubes algorithm (as described in U.S. Pat. No. 4,710,876) can,for example, be used to extract the isosurface image data output 71. Amarching cubes algorithm generates a polygonal mesh of vertices andnormals that define an outer surface of each vertebra 23.

A sample preoperative surgical plan 72 indicates two different principleaxes 26 selected according to the system 100. The cone of acceptance 25can also be included in the preoperative surgical plan output 72. Thecoordinates of the vertebrae data and principle axes 26 are known, sincethey are generated from the same preoperative CT dataset. These data aremerged to generate a combined surgical plan output 73. The combinedsurgical plan 73 has known 3D coordinates which establish the spatialrelationships between the preoperative CT dataset, the segmentedisosurfaces of the individual or multiple (select) vertebra 23, and theprinciple axes 26 of the pedicle screw insertion site(s).

Referring to FIG. 9, a polygonal mesh representing the surface of asurgical field is acquired by the digital projector 15 to output surfacetopology data of the structure of interest, which is acquired in block57.

As shown in FIG. 9, registration is then performed in block 58, whereineach of the vertebrae from the preoperative image dataset is registeredto the backscattered radiation topology data acquired in block 57. Aschematic of a sample registration process is illustrated in FIG. 13(a). Surface topology data 80 corresponding to vertebrae 23 of interestare acquired. The segmented isosurface data 71 are used as input datafor registration. Data for each vertebra 23 of interest from thepreoperative image dataset 76 are registered in block 81 tocorresponding data from the intraoperative topology image dataset 57individually.

As illustrated in FIG. 10, a particular example method that can beimplemented for the registration process 58 is based on iterativeclosest point (ICP), one of the most commonly used surface registrationtechniques. ICP is useful for relatively small transformations whenusing complex point clouds. To speed up and reduce the chance of findinga local minimum during the registration process, a rough initialtransformation (T_initial) can, for example, be used. Similarly, aninteractive point-based approach can, for example, be used to derive theinitial transformation.

For example, M (where M<5) representative points can be selected fromeach of the segmented isosurface image data (CT_MESH_OB_(—)1 . . .CT_MESH_OB_N) (where N is the number of elements segmented from thestructure and in this example, the number of vertebrae) to serve asvirtual fiducial markers in both the CT isosurface and backscatteredradiation surface topology datasets. Using a singular valuedecomposition algorithm, the M points can be brought into alignmentproviding an initial transformation for a more precise registrationusing the high resolution dataset.

Next, each of the segmented vertebrae 23 meshes noted above(CT_MESH_OB_(—)1 . . . CT_MESH_OB_N) is registered to the backscatteredradiation datasets using ICP in parallel with T_initial as an initialcondition for the transformation. The derived transformation matrices 59can be applied to the combined segmented isosurface data andcorresponding registered preoperative surgical plan to update thesurgical plan and orient the surgeon to match the structure of interestand the preoperative plan.

FIG. 13( b) illustrates updating the orientation of the surgicalstructure of interest and preoperative plan and the corresponding outputby the system 84. The transformation matrix in block 82 provides data toenable coordinate remapping in block 83 for updating a combined surgicalplan in block 73, corresponding to the immediate intraoperative locationand orientation of a surgical structure of interest. An example output84 is illustrated in FIG. 13( b). The output provides updated principleaxis 26 identifying the ideal location of a surgical interventionaldevice. Furthermore, since the preoperative topology image data arederived from the segmented image data of the preoperative CT scan, theentire target vertebrae from the preoperative CT may be displayed by thesystem 100.

To account for surgical field of view disruption during surgery, thesystem 100 can, for example, perform surface type identification toidentify clutter (as described below with reference to FIG. 16). Incases where there is insufficient data for registration, guidancefeedback pauses and resumes when there is a clearer field of view.Surface textures, or significant changes in elevation of thebackscattered radiation image data, acquired by backscattered radiationimaging can be used to identify the blockage of field of view asclutter. Similarly, the placement of the surgical tool and/or thesurgical interventional device within the surgical field of view can belocated and identified as non-clutter items and compensated for. Ifitems of clutter exist in the surgical field of view, the system 100can, for example, inform the user, for example through display 4, toremove the clutter from the line of sight. With the surface typeidentified and clutter removed, each of the segmented orthopaedicstructures can be registered to the backscattered radiation topologyscan.

Use of Confidence Criteria During Registration

Intraoperatively registering the three-dimensional surface topologyimage data to additional image data of the structured segment mayfurther include determining if registration is occurring within apre-set confidence criteria. If the criteria are not met, then seekingintervention to provide additional data to be used in intraoperativelyregistering. Intraoperatively registering the three-dimensional surfacetopology image data to additional image data of the structured segmentfurther can include registering unique features in a subset of pointsfrom the image data.

Accordingly, as a part of topology imaging and transformation, thesystem 100 can incorporate a confidence criteria component into theregistration or transformation process. FIG. 14 illustrates a method ofsurgical guidance with error checking and corrective intervention. Anerror checking module 113 and a corrective intervention module 114 canbe used to either reduce the system error, via additional input 115, orupdate the orthopedic structure and preoperative plan 60. Steps 110,111, and 112 are otherwise similar to the corresponding steps 50, 51,and 52 of FIG. 8. If the system 100 cannot resolve ambiguities to withinthe defined confidence criteria during the registration ortransformation, then the system 100 will require additional actions toresolve the ambiguity and can seek intervention 115. A prompt may beprovided to a user, for example audibly, such as a beep, or visually, bytext or flashing icon on a display 4.

The confidence criteria can, for example, be a fixed registration errorthreshold or variable registration error threshold that can be set, forexample, by the surgeon preoperatively or inter-operatively. If theconfidence criteria is not met, the surgeon may be asked to clear thefield of view (for example, move away objects situated between thecameras and the surgical field), remove debris (for example, blood andtissue resting on top of the surface of interest), adjustment of theangle of the camera 12 or other methods to increase exposure of the bonysurface of the vertebrae. As an alternative, the surgeon may be asked toidentify certain features on the acquired surface topology image data,such as the spinous and transverse processes of a vertebrae, which canact as landmarks to correct and/or improve the registration results.

For example, a registration error can be calculated as the root meansquare (RMS) of all matched point pairs used for alignment. As anexample, let P={p₁, p₂, . . . , p_(m)} and Q={q₁, q₂, . . . , q_(n)} bethe two surfaces to be registered, with m and n points respectively. Inthe context of a spinal surgery procedure, P may be the surface of thevertebrae acquired from CT, and Q may be the most recent intraoperativeoptical topology of the target of interest, or a previously acquiredoptical topology. A matched point pair is then found by locating theclosest point in P to a point in Q, along with pre-defined criteria suchas: (1) the normal vectors of the points must differ by less than 45degrees, (2) neither of the points is along the edge of the surface, and(3) is within a maximum allowed distance. Therefore, the closest pairpoint matching results in the point sets P′ εP and Q′ε Q of lengthr≦(m|n):

P′={p′ ₁ ,p′ ₂ , . . . ,p′ _(r)} and Q′={q′ ₁ ,q′ ₂ , . . . ,q′ _(r)}

Where p′_(i) and q′_(i) are the matched point pairs in the respectivesurfaces. The squared distance sd between two matched points p′_(i) andq′_(i) can then be defined as:

sd _(p′,q′) _(i) =(p′ _(i,x) −q′ _(i,x))²+(p′ _(i,y) −q′ _(i,y))²+(p′_(i,z) −q′ _(i,z))²

Then the RMS error of registration E_(RMS) for the r point pairs isdefined as:

$E_{RMS} = \sqrt{\frac{{sd}_{p,q_{1}} + {sd}_{p,q_{2}} + \ldots + {sd}_{p,q_{r}}}{r}}$

P′ and Q′ are used by the registration algorithm (for example, theaforementioned ICP algorithm) to determine a coordinate transform thatrigidly moves P towards Q. Then, with the updated coordinates, a new setof matched point pairs can be calculated for the next registrationiteration.

The error correction method can continue iteratively until sufficientconvergence has been achieved. For example, in each iteration, theregistration error is calculated. The i^(th) RMS error is labeled asE_(RMS) _(i) . The registration algorithm then iterates until the changein RMS registration ΔE_(RMS) is less than a threshold E_(thres), such as0.1 mm.

ΔE _(RMS) =|E _(RMS) _(i) −E _(RMS) _(i-1) |<E _(thres)

The final registration error may then be displayed to the surgeon, wherehis/her approval may be requested to proceed with navigation.

FIGS. 18 (a)-(c) illustrate typical results 300 of this example method,demonstrating the iterative registration error process with theconvergence of one registered optical topology dataset to a subsequentoptical topology dataset, with the user selecting a confidence criteriaof E_(thres)=0.1 mm. The resultant E_(RMS) and ΔE_(RMS) values aredepicted in Table 1 below:

TABLE 1 Example results of registration error calculation, topology totopology, representing quantitative results from the exampleregistration error calculation shown in FIG. 18. Iterations 1 2 3 4 5E_(RMS) _(i) (mm) 5.67721 2.46665 0.344543 0.11417 0.113973 ΔE_(RMS)(mm) — 3.21056 2.122107 0.230373 0.000197

Furthermore, FIG. 19 demonstrates that this iterative registration errorconverges to the pre-defined confidence criteria of one optical topologydataset to CT surface registration, with the user selecting a confidencecriteria of E_(thres)=0.1 mm. The resultant E_(RMS) and ΔE_(RMS) valuesare depicted in Table 2.

TABLE 2 Example results of registration error calculation, topology toCT, representing quantitative results from the example registrationerror calculation shown in FIG. 19. Iterations 1 2 3 4 5 6 E_(RMS) _(i)3.70133 3.53156 3.06149 1.47036 0.823002 0.776784 (mm) ΔE_(RMS) —0.16977 2.16997 1.59113 0.647358 0.046218 (mm)

Referring now to FIG. 15, an example method is shown in which onlybackscattered radiation image data previously registered to thepreoperative image data is registered. The previously registeredtopology data is segmented in step 123, where clutter items may beremoved, as they do not contribute to the registration of desirable bonysurfaces. This produces a segmented optical data subset in step 124. Inthis method, for example, when counter 126 i==1, the registration 58proceeds as in FIG. 8. After the initial registration 125 (with thecounter 126 i>1), each of the preoperative image data structure segments(CT_MESH_OB_(—)1 . . . CT_MESH_OB_N) is registered to the backscatteredradiation topology dataset, and only the registered surfaces areretained in the backscattered radiation topology dataset. Again, steps120, 121, and 122 are otherwise similar to steps 50, 51, and 52 of FIG.8.

This process generates a segmented backscattered radiation topologydataset comprising the exposed structures of interest (OT_MESH_OB_(—)1 .. . OT_MESH_OB_N) and excludes non-relevant targets, such as surgicaldrapes. This dataset is then used in the next registration iteration asa proxy for the segmented orthopaedic structures dataset. As theregistration is limited to only the relevant exposed surfaces, thebackscattered radiation image dataset can be reduced in size so thatregistration can be faster.

In one example method, the initial registration can be performed whenthe structure is sufficiently exposed to the camera 12 and allobstructions are removed between the exposed structure and the camera12. The initial registration can be repeated if there is a change to theregion of interest intraoperatively, for instance, the field of view ischanged to a different vertebra, or there are changes in the anatomy dueto surgical intervention,

Texture-Based Surface Identification for Clutter Rejection

In one embodiment, the system performs surface type identification toidentify and remove clutter. This step can be useful in detecting,accounting for, and optionally correcting field of view disruption. Suchfield of view disruptions can occur during operation due to movements bythe surgeon or other personnel. In cases where there is insufficientdata for registration, guidance feedback pauses and resumes when thereis a clearer field of view. Referring to FIG. 16, an example embodimentof surgical guidance with surface identification 133 and a clutterremoval processing step 134 in block 102 is illustrated. Blocks 130,131, and 132 of FIG. 16 are otherwise similar to blocks 50, 51, and 52of FIG. 8.

In one example, surface textures, or significant changes in the centerof mass of the backscattered radiation image data, acquired bybackscattered radiation imaging, can be used to identify the blockage offield of view as clutter. Similarly, the placement of surgical tools andimplants within the field of view can be located and identified asnon-clutter items and compensated for. If items of clutter exist in thefield of view, implementations of the system 100 can, for example,inform the user to remove the clutter from the line of sight.

In addition, color textures acquired by the camera 12 can be employed todifferentiate structures within the backscattered radiation image dataof the operative region of interest to provide additional informationfor improving the speed and accuracy of the registration/transformation.For example, and as further described below, non-clutter, non-structuresurface can be ignored during registration and transformation ofbackscattered radiation image structure surface.

Spectral rejection of clutter can be employed by recognizing thatdifferent materials in the field of view scatter more or less of someportions of the electromagnetic spectrum. A potential way to improvesurface identification is through the use of the filter(s) 40 integratedwith the camera 12 to preferentially accept only certain bands of theelectromagnetic spectrum. Such filters can be optimized to achievemaximum contrast between different materials and thus improve theclutter identification process. For example, those bands that are commonto backscattered radiation from typical clutter items, the structure,and surgical tools can be filtered out such that high-contrastbackscattered radiation is acquired.

In orthopaedic applications where it is usually the case that the bonysurfaces from the topology image sets are to be registered to thepreoperative CT or MRI data, it may be effective to first removenon-bony-surfaces such as soft tissue, surgical drapes, and otherirrelevant surface that are commonly observed during a surgicalprocedure. A number of methods could be used independently, or inconjunction with each other, to achieve this goal. The followingparagraphs describe specific implementations and examples of potentialclutter rejection algorithms based on spectral-based rejection,color-based rejection and surface roughness-based rejection.

Spectral-Based Clutter Rejection

Structured light illumination is typically performed with white light.However, if one or more specific spectral bands are employed to performthe structured light illumination, certain spectral regions can berejected. This in turn can be employed to eliminate certain surfacesfrom the acquired image data due to the specific absorption andscattering properties of various materials. For example, high absorptionand or low scattering within the implemented spectral band will limitthe visibility of the low scattering region to the camera(s).

FIG. 22 demonstrates an example implementation of this method, whereimage 340 is obtained according to structured light reconstruction usingwhite light illumination, 341 is obtained according to structured lightreconstruction using red light illumination, and 342 is obtainedaccording to structured light reconstruction using green lightillumination. The majority of the surface is white (W) in color with twosmall regions of red (R) and green (G). Under white light illuminationall three regions are captured and reconstructed. Under red illuminationonly, the white and red surfaces are captured and reconstructed.Moreover, under green illumination, only the white and green surfacesare captured and reconstructed.

Accordingly, an example implementation of the present spectral basedclutter rejection technique could include the automatic identificationor removal of specifically colored tools, gloves, drapes, etc., withinthe surgical field of view. Alternatively, white light illuminationcould be used, where band pass filters 40 within the field of view ofthe cameras could be used to image specific spectral bands of interest.It will be apparent to those skilled in the art that there are a widevariety of methods for achieving spectrally selective detection,including employing spectrally narrow emitters, spectrally filtering abroadband emitter, and/or spectrally filtering a broadband imagingdetector (e.g. camera).

FIG. 23 demonstrates an example implementation of how colorized meshdata 350, acquired with a structured light scanner, can be employed toreject muscular tissue, while maintaining bony surfaces 351. Monochromecameras may be employed to reconstruct a 3D surface using structuredlight imaging. However, the use of color cameras allows for the directassignment of color values to each point of the reconstructed surface.These color values are stored as tuples of RGB values (i.e. (R,G,B),where R,G,B are elements of {0-255}) stored at each mesh point, 350. Thealgorithm then traverses the mesh and generates a set of seed points,for example, with a spacing of Δr=1.0 mm. Next, at each point in themesh the R/B and R/G ratio values are calculated. Taking the ratio of R,G and B values, instead of directly using raw values, provides a methodto help mitigate effects induced by variable illumination.

Alternatively, more complex methods can be applied to better deal withillumination variability, for example, as taught in Lin et al. (C. Linet al, “Color image segmentation using relative values of RGB in variousillumination conditions” International journal of computers Issue 2 Vol5, 2011). Average R/B and R/G values in a disk, with radius r=2.0 mm,surrounding each seed point are then calculated. Finally, these disksare rejected or accepted based on their average ratio values (similarly,green and blue surfaces could be rejected by setting threshold valuesfor G/R, G/B and B/R, B/G respectively). For the specific example inFIG. 23, regions that fulfilled the criteria of 0.9<R/G<1.4 and0.9<R/B<1.4 produced an image 351, where bony surfaces were identified,while muscle was removed from the resultant dataset.

Surface Roughness-Based Clutter Rejection

In another embodiment, clutter rejection is performed using detectedvariations in surface roughness, where the variations are detected usinga surface topology backscattered radiation image acquisition system 1.In an example implementation, mesh data acquired with a structured lightscanner can be employed to reject muscle tissue, while keeping bonysurfaces in the topology dataset. The basis for this algorithm is theknowledge that most bony surfaces are relatively smooth, while musclehas a striated structure. In addition, the muscles are subject tocutting during spinal surgery by the surgeon, further contributing totheir surface unevenness. Combined, this gives rise to large curvaturesin the mesh that may be detected. This can be characterized bycalculating the maximum principal curvature at each point in the mesh,for example, as shown in Guggenheimer et al. (Guggenheimer, Heinrich“Chapter 10. Surfaces”. Differential Geometry. (1977) Dover), which inturn can be used to reject the muscle tissue when compared to a bonysurface. The clutter rejection process begins by acquiring an opticaltopology scan, after which a surface roughness-based clutter algorithm,optionally executed by surgical guidance controller 3, calculates themaximum principal curvature at each point in the mesh. The algorithmthen traverses the mesh and generates a set of seed points, for example,with a spacing of Δr=1.0 mm. The maximum principal curvatures are thenaveraged in a disk, for example, with radius r=2.0 mm, surrounding eachseed point. Finally, the disks are accepted or rejected based on theaverage curvature values.

The resulting clutter-based rejection is illustrated in FIG. 24, where360 shows structured light reconstruction of bone and muscle tissueunder white illumination, and where 361 shows structured lightreconstruction of bone and muscle tissue under white illumination withroughness based clutter rejection. For example only regions thatfulfilled the criteria of a curvature less than 0.7 were kept to produceimage 361 from image 360 in FIG. 24.

With the surface type identified and clutter removed, each of thesegmented orthopaedic structures can be registered to the backscatteredradiation topology scan, potentially with greater accuracy.

Continuous System Operation

In one embodiment, system 100 may act autonomously for intermittentperiods of time, such as performing regular updates of the imageregistration and/or surgical plan without user input. In one example,the system may provide an external request for user action to enable thesystem perform semi-autonomously under circumstances where insufficientimage data is available in the field of view. For example, the systemmay provide the user with continuously updated surgical guidancefeedback in the form of an image of the current orientation of thesurgical structure of interest outputted on the display, and updatedsurgical guidance plan for the accurate placement of the surgicalinterventional device. However, in the event that the surface to beimaged is obscured or blocked, for example, by a surgeon's arm, thesystem may alert the user and temporarily suspend image registrationprocessing and displaying of results.

Continuous updating of surgical guidance feedback may occurautonomously, such that upon completion of one update, another updateautomatically commences. In such an embodiment, the user is not requiredto manually input a request for an update from the system 100.Accordingly, the use of system 100 may be advantageous in reducingsurgical procedure time, due to real time updates of the surgicalstructure of interest, compared to other systems currently in use.

In one example, the rate of data updating may be contingent or dependenton a temporal margin for error at any given time point during a surgicalprocedure. The temporal margin for error may be dependent on the timerequired to achieve a potential negative outcome, for example, thesituation where the surgeon is not operating at an ideal targetimplantation site. The time required for the system to achieve thepotential negative outcome may be a factor of the accuracy of thesystem, the spatial margin for error at a given time in a givenprocedure, and the speed at which the procedure is being performed.

For example, if a clinician has 5 mm of spatial margin and the system isaccurate to within 2 mm of the ideal interventional device implantationlocation, then the spatial error margin is 3 mm. If the clinician ismoving at 1 mm per second or less, then the clinician has 3 seconds oftemporal margin for error. Updates could occur continuously in thisscenario once every 3 seconds to avoid an error. Any error in thecalculation of an implantation trajectory at a given time may not leaddefinitively to a negative outcome, however, such an error can reducethe margin for error for future points along the principle axis.Accordingly, more frequent updates will lead to improved feedback andaccuracy. Depending on the execution speed of the surgeon (typicallyslow for precise procedures), multiple updates per given unit time(i.e., one or a few seconds) may provide the appearance of continuousmotion without stutter for image-based guidance feedback. For text-basedsurgical guidance feedback, updates may need to be slower to allow oneupdate to be read before the next occurs.

In one embodiment, one or more fiducial markers may be attached to, orworn by, the surgeon in order to dynamically determine the rate ofchange of motion of the surgeon. The rate of change of motion may bemeasured by the cameras of the backscattered radiation surface topologyimaging device, or may be detected using a global position sensingsystem (described further below). The rate of system updating may thenbe dynamically adjusted according to the measured rate of the surgeon'smovement. In another example, the rate may be determined by dynamicallymeasuring the rate of change of the position of a tool, probe or othersurgical instrument to which fiducial markers are attached.

Tool Tracking

In selected embodiments, a tool, such as a surgical tool, probe,surgical instrument, or other freely movable item, having fiducialmarkers adhered thereto, may also be integrated with and tracked bysystem 100, in order to co-register the position of the tool with 3Dpre-operative image data. Tool tracking can be performed via severaltechniques, such as passive infrared reflectance triangulation or activeemitting triangulation. The surgical tool can be tracked to provide asurgeon with a feedback mechanism (i.e. visual or audio or both) toidentify the planned trajectory (x, y, z, roll, yaw, pitch) of thesurgical tool. Such guidance feedback can assist in accurate deviceplacement. The system 100 can, for example, also track the position ofthe implantation device until it reaches a planned location or depth tofurther assist in accurate device placement.

FIG. 25 demonstrates an embodiment showing the integration of tooltracking with a surface topology imaging system, which includesprojector 15 and camera 12 to yield a complete surgical navigationsystem. The spatial locations of the projector 15 and cameras 12, alongwith the surgical tool 6, can be computed via triangulation of fiducialmarkers 371 as detected by optical position measurement system 370. Theidentification of these fiducial markers can occur using varioustechniques, where common methods include passive IR ball tracking oractive emitting technologies.

FIG. 26 summarizes the relation between the coordinate systems relevantto FIG. 25. The location and orientation of surgical target 8 (forexample, a vertebra of interest) relative to the surface topologyimaging system 373 is known based on the optical topology measurements.The location and orientation of the surface topology imaging system 373relative to an optical position measurement system 370 (such as one forthe purpose of tool tracking) is known via the detection of the fiducialmarkers on imaging system 373 as detected by optical positionmeasurement system 370. The combination of these two pieces ofinformation, allow the topology data to be registered into thecoordinate system of the position measurement system 370. Finally, thelocation of the surgical tool 6 relative to the optical positionmeasurement system 370 is known, similarly via the detection of thefiducial markers on tool 6 as detected by optical position measurementsystem 370. Therefore, the location of the surgical tool 6 relative tothe surgical target of interest 8 is now defined as both the surgicaltarget and tool are tracked in the coordinate system of positionmeasurement system 370.

With the positional information of the tool 6 relative to the vertebraeknown, the cone of acceptance 25 and current surgical tool spatiallocation 6 are displayed 8 to the surgeon, via a portable workstation 7to provide real-time feedback to aid in the placement of interventionaldevices (i.e. pedicle screw, rod, etc). As an example, the topologyprojector system 373 can be attached to the surgical table 374 andpositioned into an appropriate imaging position via a reticulating arm375. The topology projector system 373 can also be positioned on eitherside of the surgical table 374 to provide an optimal imaging field ofview. Alternatively, the topology projector system could also beattached to a portable cart, be ceiling mounted or attached to thesurgical room lighting system to achieve an optimal surgical field ofview.

Another example implementation of the surgical navigation used in theoperating room by surgeons 390 is shown in FIG. 27. The topology imagingsystem is a handheld device 391 with field of view 392 overlooking anincision 393 made on the patient 394 to expose the spine. The patientrests on top of a positioning couch 395 on a surgical table 396. Unlikethe example system illustrated in FIG. 25, where topology projectorsystem 373 was mounted to surgical table 374 and reticulating arm 375,in the present example, topology projector system 373 may be freelymoved intraoperatively. The ability to decouple topology projectorsystem 373 from a rigid frame arises from the placement of the fiducialmarkers 371 on topology projector system 373, thereby enabling thespatial tracking of both topology projector system 373 and tool 6 in acommon, global reference frame, by optical position measurement system370. In order to compensate for the motion of both topology projectorsystem 373 and tool 6, system updates may be performed periodically on asuitable timescale (as described above).

Accordingly, in the present example, as illustrated in FIG. 25, asurgical guidance system is provided, in which a fiducially markedsurface topology backscattered radiation image acquisition system and afiducially marked tool are freely movable relative to each other, andrelative to the patient, by virtue of the their positional detection andreference frame registration using optical position measurement system370. The present system thus provides a surgical guidance system wherebyall fiducial marking have been effectively transferred from the patientto the system. Moreover, the present system does not requireintraoperative recalibration. This embodiment avoids requiring acalibration step to register topology projector system 373 to tool 6,thereby saving time and positively impacting clinical workflow.

Another example system is provided in FIG. 28( a). Unlike the exampleprovided in FIG. 27, the present example implementation utilizestopology projector system 373 for measuring both the surface of intereston the patient and for triangulation-based tool tracking. By employingtopology projector system 373 for both functions, the system is operablewithout the need for optical position measurement system 370.

Referring to FIG. 28( a), the example system 400 includes a frame 401supporting two cameras 402 equipped with optical filters 403. Cameras402 detect fiducial markers 404 adhered to tool 405. Fiducial markers404 may be passive reflective markers, or active emitters, provided thatlight emitted or reflected by fiducial markers 404 is detectable bycamera 402 after passing through filters 403.

Fiducial markers 404 are illuminated by source 406, which may be any ofa wide range of optical sources. If fiducial markers 404 are passivelyreflecting markers, then source 406 has a spectral profile that ischosen to be transmitted through filter 403. Alternatively, if markers404 are fluorescent markers, then source 406 is selected to have aspectral profile suitable for generating fluorescence from markers 404,and filter 403 includes a spectral passband for transmitting the emittedfluorescence.

In one example, fiducial markers 404 are passive infrared (IR) balls. IRlight for illuminating passive IR balls 404 attached to the tracked tool405 is provided by source 406. Source 406 is shown in the example figureas light emitting diodes.

System 400 is characterized by field of view 407, which is determined atleast in part by the angular emission bandwidth of light source 403 andthe angular acceptance bandwidth of cameras 402. During operation, frame401 is oriented such that field of view 407 envelops the surface ofinterest on the surgical target.

Topology information is obtained by topology projector system 373, whichmay be a structured light source including, for example, projector 408for illuminating the target for topology imaging. Projector 408 may be aminiature projector. In order to utilize cameras 402 for both topologydetection and tool tracking, the emission spectrum of topology projectorsystem 373 is selected to support detection of backscattered radiationby cameras 402. This is achieved by selecting a spectrum of filter 403and/or an emission wavelength of topology projector system 373 such thatthe backscattered radiation passes through filter 403. In one example,the bandwidth of filter 403 is chosen to transmit both the backscatteredradiation and the optical signal provided (for example, reflected oremitted) by fiducial markers 404. In another example, filter 403 may becharacterized by multiple optical passbands for transmitting both thebackscattered radiation and the optical signal provided by fiducialmarkers 404. In another example, two filters may be provided andperiodically translated or rotated into the optical path of camera 402,where each filter is provided for a separate imaging modality(topological detection and fiducial marker detection).

It will be understood that cameras 402 and projector 408 each have anassociated focal region extending over their respective depths of field.In one embodiment, cameras 402 and projector 408 are configured suchthat their respective focal regions overlap at or near a given region ofinterest. The overlapping of the two focal regions (such that bothsystems overlap over at least a portion of their respective depths offield) enables accuracy in terms of both the resolving of the finespatial features by cameras 402, and in terms of projecting thefinest/clearest patterns by projector 408. This overlap can be obtained,for example, by appropriate selection of focusing or beam conditioningoptics, by the relative positioning of the two subsystems, and/or therelative orientation of the two subsystems, such that the respectivefocal regions overlap at or near the desired region of interest.

In some embodiments, the focal regions may be overlapped such that theregion of overlap is of sufficient distance away from system 400 so asto not obstruct or otherwise impair the surgeons during a medicalprocedure. For example, in some example implementations, this distancemay be in the range of approximately 1.5 to 3 meters. Therefore, in oneexample implementation, system 400 may be positioned so that it isapproximately 1.5 to 3 meters away from the operating region ofinterest, and such that the overlapping focal regions of cameras 402 andprojector 408 are at the operating region of interest. It is to beunderstood that the field of view of illumination sources 406 shouldalso overlap this region of interest, in order to provide suitable andsufficient illumination of fiducial markers 404.

For simultaneous real-time triangulation-based tool tracking andtopology imaging, the system 400 may be controlled such that imageacquisition is configured for supporting both imaging modalities. In oneexample, in which cameras 402 are employed for both imaging modalities,the detection of surface topology via backscattered radiation and thedetection of the position and orientation of tool 405 may be performedserially. For example, the two modalities may be interleaved such thecameras 402 acquire a first set of images when only topology projectorsystem 373 is active (i.e. emitting light), and subsequently a secondset of images is acquired when only the tool tracking light source 406is turned on, where the process is thereafter repeated (for example, ona continuous basis).

This serial acquisition method is illustrated in the flowchart providedin FIG. 29 for the example case of a structured light system. In step500, the structured light projector is activated. The surface ofinterest is illuminated with optical fringe patterns in step 510, andthe topology image is detected and processed. Subsequently, in step 520,the projector is deactivated and the optical fringe pattern is no longerprojected onto the surface of interest. In step 530, the tool trackinglight source is activated, and the signals from the fiducial markers aredetected and processed in step 540. Finally, in step 550, the tooltracking light source is deactivated, and the process may be repeated.The number of acquisitions n and m can be varied, depending on thetemporal and signal-to-noise requirements of tool tracking vs. topologyimaging.

In order to display the tool position and orientation with pre-operativeimage data co-registered to the surface topology images, the referenceframe of the tool tracking system is registered to the reference frameof topology projector system 373. Since the system frame 401 houses boththe topology imaging and triangulation-based tool tracking hardware, thelocation of the surgical tool 405 relative to the imaged vertebral bodyof interest can be established through a calibration procedure. Thisprocedure only needs to be performed once, provided the position of thecameras and projector are not altered in the system fixture.

In the above example, an integrated, interframe system is described inwhich cameras 402 are employed for the dual role of detecting topologysurface signals from the topology projector system 373 and detectingpositioning signals from fiducial markers residing on the tool. Inanother embodiment, a second set of cameras may be provided, such that adedicated set of cameras are provided for each imaging modality. Thisexample implementation relaxes the optical system requirements and maybe useful in enabling the use of dedicated cameras for each imagingmodality that are suited to the needs of each modality, and also mayprovide a system with a faster response rate by avoiding the need toserially operate the cameras. Such embodiment is shown schematically inFIG. 28( b). System 420 is similar to that of the system 400, with theaddition of two cameras 429. As before, cameras 402 detect fiducialmarkers 404 adhered to tool 405, where the fiducial markers 404 areilluminated by source 406. To acquire surface topology information,projector 408 is used to illuminate the target. The backscatteredradiation from the projector is then detected by cameras 425. Opticalfilters 430, placed in front of cameras 425, are selected that has apassband in the range of wavelengths from projector 408, but that doesnot overlap significantly with the passband of filters 403.

Similarly, in some embodiments, cameras 402, 425, and projector 408 maybe configured such that their respective focal regions (corresponding totheir respective depths of focus) overlap at or near the region ofinterest, as described above with regard to FIG. 28( b). As describedabove, the field of view of illumination sources 406 should also overlapthis region of interest, in order to provide suitable and sufficientillumination of fiducial markers 404.

In system 400 and 421, the cameras and projection systems are housed ina rigid frame 401, therefore a fixed relationship exists between each ofthe components. Similarly, a fixed relationship exists between thecoordinate system used by the surface topology imaging system and thecoordinate system of the tool tracking system. This relationship existsin the form of a calibration bringing one of the two coordinate systemsinto the other.

In one embodiment, the calibration between the surface topology imagingdevice and the position measurement device can be determined by imaginga set of targets visible by both devices, for example, reflective IRspherical markers. Using the acquired surfaces from the surface topologyimaging datasets and the centroid positions of spheres from the positionmeasurement device, a set of common points and/or features can beextracted. In the case of position measurement device, the centroid ofeach sphere is directly available. In the case of the surface topologyimaging device, the center of each sphere can be extracted using themethod disclosed below using back projection of surface normals. Atransform between the two datasets can then be calculated using aregistration routine such as a landmark transformation which calculatesthe best fit mapping of one point set onto the other, in a least squaressense. A minimum of three data points in each point set are required.

FIG. 30 is a flowchart illustrating an example method of performing thecalibration with IR balls. The first steps 451 consist of positioningthe topology imaging device, tool tracking device, and the IR spheressuch that they are stationary relative to each other. The second steps452 involve obtaining the centers of the IR spheres from the twosystems. For the segmentation step, this can be done manually. The thirdsteps 453 performs a registration on the two point sets to calculate thecalibration transform that brings one of the two coordinate systems intothe other.

Surface Identification and Tool Tracking

In another example embodiment, tool tracking may be directly integratedwith the topology projector system such that the position andorientation of a tool is detected using the same topological imagingmodality that is employed to optically interrogate the surface ofinterest. For example, surface identification may be employed to trackthe location and orientation (e.g. the pitch) of a surgical tool.Accordingly, the ability to provide surgical guidance feedback fororthopaedic structures as described previously can be enhanced withtopological-based guidance feedback relating to the surgical tool in 3Dspace, for example, by depicting where the surgical tool is and where itis planned to be with respect to the 7 degrees of freedom (x, y, z,roll, yaw, pitch, time) and may additionally be used for the placementof surgical tools or other items of interest.

Referring to FIG. 17, an example embodiment of surgical guidance withsurface identification 133 and tool tracking 143 is illustrated. Block140, 141, and 142 are otherwise similar to Block 130, 131, and 132 ofFIG. 16. This embodiment is further illustrated in the example below,with reference to FIGS. 30 to 32.

In the present non-limiting example, the tracking of a surgical tool isillustrated using surface identification via a structured light system.FIG. 31( a) shows an image of a tool 600 to be tracked. In a first step,fiducial markers are attached or adhered to the tool. The fiducialmarkers are passive, surface identification based, markers that areselected to be identifiable by the structured light system at multiplepositions and orientations. Accordingly, in one example, the markers maybe spherical balls 605, which present a common surface profileindependent of orientation.

In one example implementation, the balls may have a diameter of 0.5-1cm. Smaller ball diameters may also be employed using a camera ofsufficient resolution and an appropriate fringe projection. Increasingthe resolution of the system generally requires more computation powerto process the data. It is to be understood that alternativenon-spherical surface profile markers may alternatively be employed,such as planar polygon shapes (for example, triangles and squares),where the corners of the polygon can be used to determine the center ofthe shape.

In general, any landmark on a tool can be specified as a fiducialmarker, provided that a suitable surface can be identified over a widerange of positions and angular orientations. Also, these landmarksshould be sufficiently spaced spread out across the tool of interest toincrease the tracking accuracy. In practice, these landmarks should notbe positioned so that they can be blocked from the field of view whenthe tool is held. As will be shown below, a sphere is an effectivemarker since the center of a sphere can be easily extracted even if itis partially blocked.

The marker balls may be attached or adhered to the tool at three or morelocations to support 3D position and orientation sensing. In oneexample, the marker balls may be screwed onto the tool at 3 locations.Other techniques of attaching the balls include snap-on methods orpermanent attachment via adhesives, depending on the required use of thetool.

After having attached the markers to the tool, a 3D surface model of thetool is obtained. An orientation axis 610 and tip position 615 of thetool is then determined and recorded. The 3D model can be imported fromcomputer aided design (CAD) drawings, in which case the tool's tip,orientation axis, and the position of the marker balls can be specified.Alternatively, for a tool without a CAD drawing, the tool can beprofiled with the structured light scanner to obtain its 3D geometry.From the acquired point cloud, the tool tip and orientation axis can beextracted. The center of the marker balls can then be estimatedmathematically based on the shape of the marker balls or specifiedmanually.

For simple cylindrically symmetric shapes (e.g. a cylinder), theorientation axis and tip position may be calculated automatically basedon the dimensions of the tool, using either the CAD drawings or from thepoint cloud acquired from a structured light scanner. In another examplemethod, the orientation axis and tip may be defined manually via CADdrawings.

In the case where the tool does not have a tip or is not cylindricallysymmetric, different measures can be used to specify the position andorientation of the tool. For example, in the case of an ultrasoundtransducer which is being tracked, the center of the transducer can bespecified and an orientation axis can be defined by first defining aplane tangential to the transducer face. Then a normal to this plane,which passes through the center of the transducer, can be used tospecify an orientation axis. For surgical navigation, the orientationaxis is generally centerline of the tool, as it generally aligns withthe axis of an interventional device to be inserted, such as a screw.

The orientation axis 610 and tip position 615 can be stored as 3D vectorand a point relative to the coordinate system of the CAD drawing orpoint cloud. This can be saved into a calibration file that can beloaded into the navigation system. Multiple calibration files may begenerated and loaded such that multiple tools can be trackedindividually or at the same time during navigation.

The center of the marker balls to be tracked is then determined andrecorded in a relative coordinate system. These centers are denoted inthe present example by {P1, P2, P3}, and the center of one of the ballsis shown in FIG. 31( b) at 620. This specifies a unique geometry thatwill be isolated and tracked intraoperatively, as shown below. Thesethree points uniquely determine the orientation and location of the fulltool in 3D space and hence the orientation axis and tip position.

The topology of the tool is then intraoperative scanned using thestructured light projection system. In order to detect the position andorientation of the tool, three marker balls should be partially visiblein the scan. FIG. 32( a) illustrates an acquired surface topology scanof the tool and the three marker balls. As can be seen, the full tooldoes not need to be in the field of view and there can be additionalsurfaces in the field, such as the surgeon's hands. If, during tracking,shadowing occurs such that two or less balls are detected, the trackingis stopped. Such an event can be limited through the positioning ofmultiple cameras such that the markers are consistently in the field ofview. Alternatively, there can be more than 3 markers on the surgicalinstruments, to increase the probability of any 3 markers being visibleat any given time.

After obtaining the topological surface scan, the surfaces of the ballsare identified. In one example, the surfaces are identified usingspectral filtering to isolate the ball surfaces from the remainder ofthe image. This may be performed by acquiring a full color (white light)surface profile of the tool using a standard projector and a colourcamera. By using ratios of the R, G and B channels, the marker ballssurfaces 625 can be identified as seen in FIG. 32( b). In the presentexample, ball surfaces were identified by G/R and G/B values greaterthan 1.1 and less than 255. No filters were used in this implementationand all points outside of this range were removed. In an alternativeembodiment, bandpass filters could be used to accomplish similarresults.

Having identified the ball surfaces, the ball center locations {Q1, Q2,Q3} are then determined. This can be accomplished by back projection ofsurface normals and determining the closest point of approach for eachpair of normals. The mean of all closest points of approach for eachmarker ball is the approximated center. This method is illustrated inFIG. 33( a), where normals 630 and 640 are employed to determine centerlocation 645 of a ball. In principle, only two normals are needed tospecify an approximate center of the marker ball. However, FIG. 33( b)demonstrates how the standard deviation of the point location decreasesas the number of normal pairs used increases. A smaller standarddeviation results in a smaller variability in locating the center of themarker ball.

Finally, landmark registration of the ordered sets {P1,P2,P3} to{Q1,Q2,Q3} may be performed to obtain a transform M that maps the fulltool to the partial surface scan of tool. Methods for landmarkregistration of ordered sets are known to those skilled in the art, forexample, as described in Berthold K. P. Horn (1987), “Closed-formsolution of absolute orientation using unit quaternions”. Havingdetermined the transform M, the full tool model is then transformed tothe current tool position. As shown in FIG. 34, the full tool may besubsequently shown in display, including known 3D surface based oninitial measurement, and the system can track the spatial position andorientation of surgical tool using surface-based registration. Theresulting tracked tool may be displayed with co-registered 3Dpre-operative image data and/or a surgical plan, as described in thepreceding examples.

The present spectral identification method may be extended to enable thesimultaneous tracking of multiple tools, provided that the balls on eachdifferent tool have different colors. This can be achieved by selectingmaterials that can be identified throughout the any suitable portion ofthe electromagnetic spectrum, such as UV, visible, IR, etc. Preferably,ball marker colors are selected that would not typically be found in thesurgical field of view.

The preceding description has been made with respect to the provision ofguidance feedback to surgeons, however, it is recognized that suchguidance feedback can also be provided to, and utilized by, otherpersons or systems, such as autonomous or semi-autonomous surgicalrobotic systems for the automated guidance of such surgical roboticsystems. Furthermore, although many of the preceding examples includethe co-registration of a surgical plan, it is to be understood thatembodiments may be practiced without the incorporation of a surgicalplan.

Furthermore, while the preceding disclosure has been provided in thecontext of surgical navigation, it is to be understood that the scope ofthe embodiments provided herein not intended to be limited to surgicalpractice. Examples of implementation embodiments as described above areprovided for illustration purposes rather than to limit the scope ofpossible embodiments. Accordingly, systems and methods disclosed hereinmay be adapted to a wide variety of uses and applications in which it isuseful or desirable to employ the registration of surface image data tothree-dimensional volume image data.

For example, the embodiments provided herein may be useful in fieldssuch as test and measurement, manufacturing, non-destructive testing,geo-prospecting, training, education, mixed reality applications and thevideo game industry. In a manufacturing example, made products could bedimensionally compared and quantified to their original computer aideddesign (CAD) to verify a proper design and manufacturing processes viathe system described herein. An additional manufacturing applicationincludes use in an assembly line, where components are added to a basestructure. Topology imaging of the base structure can be used toidentify its position and orientation. Similarly, the robotic arm'sposition and orientation can be tracked. Using the present method, thiswould allow precise placement of components onto the base structure viathe robotic arm. Another application is the identification ofinefficient machining tools in a computer numerical control (CNC)system. The individual machine bits of a CNC machine are routinelychanged when they become dull or broken. The system described hereincould create 3D profiles of all the machine bits, prior to or duringsystem use, for comparison with pre-loaded ideal bit profiles. Thesystem could then register the 3D bit profiles, to the pre-loaded modelto identify bits that have become dull, have been installed improperlyor have broken in an effort to reduce machining errors. The methodsdisclosed can also be used to identify or sort through items in anassembly line, where the topology of the item under inspection can becompared to a known model of the item.

An additional example includes the use of the system described herein totrack multiple targets using surface type identification to mergevirtual models of human actors, animals, vehicles, etc. for the videogame or computer generated imagery industry. The present embodiments canalso be of use in mixed reality applications for surgical training. Forexample, the position and orientation of a patient phantom can bedetermined using its 3D topology. Through augmented reality using headmounted displays, or other forms of displays, that are tracked in space,different clinical scenarios can be overlaid onto the patient phantom.Physical tools held by the trainee would be tracked relative to thepatient phantom, to allow interventions to be performed virtually. Incertain scenarios of the above examples, portability of the system maybe necessary for field use. Simultaneous real-time triangulation-basedtool tracking and topology imaging (system 400) may be advantageous.Such portability may be suitable to be fitted onto a mobile robot, toperform object identification to navigate a terrain, and perform objectmanipulation through a tracked robotic arm.

Although some of the drawings illustrate a number of operations in aparticular order, operations which are not order-dependent can bereordered and other operations can be combined or broken out. While somereordering or other groupings are specifically mentioned, others will beapparent to those of ordinary skill in the art and so do not present anexhaustive list of alternatives. Moreover, it should be recognized thatthe stages could be implemented in hardware, firmware, software or anycombination thereof.

In various embodiments, hardwired circuitry can be used in combinationwith software instructions to implement the embodiments. Thus, thetechniques are neither limited to any specific combination of hardwarecircuitry and software nor to any particular source for the instructionsexecuted by the data processing system. In this description, variousfunctions and operations are described as being performed by or causedby software code to simplify description. However, those skilled in theart will recognize what is meant by such expressions is that thefunctions result from execution of the code by a processor, such as amicroprocessor.

The specific embodiments described above have been shown by way ofexample, and it should be understood that these embodiments may besusceptible to various modifications and alternative forms. It should befurther understood that the claims are not intended to be limited to theparticular forms disclosed, but rather to cover all modifications,equivalents, and alternatives falling within the spirit and scope ofthis disclosure.

What is therefore claimed is:
 1. A surgical guidance system comprising:a storage medium for storing pre-operative image data associated with apatient; a surface topology imaging and optical position measurementsystem comprising: an optical projection device configured to projectoptical radiation onto an exposed surface of the patient, such thatbackscattered optical radiation is suitable for optical surface topologydetection; an optical source having a wavelength selected to illuminatea set of fiducial markers provided on a movable instrument; two or morecameras, wherein at least one of said two or more cameras is configuredfor imaging the backscattered optical radiation, and wherein at leasttwo of said two or more cameras are configured for imaging said set offiducial markers when illuminated; and a surgical guidance controlleroperatively connected to said surface topology imaging and opticalposition measurement system and said storage medium, wherein saidsurgical guidance controller includes a processor configured to: controlsaid optical projection device to illuminate the exposed surface andobtain, based on one or more images from said at least one camera,topological image data associated with the exposed surface; and controlsaid optical source to illuminate said set of fiducial markers andobtain, from said two or more cameras, positional image data associatedwith said set of fiducial markers; determine a position and anorientation of said movable instrument relative to the exposed surface;and register said topological image data, and said position andorientation of the movable instrument to said pre-operative image data;wherein said surgical guidance controller is configured to control saidoptical projection device and said optical source such that operation ofsaid optical projection device and said optical source are temporallyinterleaved, and such that the backscattered optical radiation is imagedduring operation of said optical projection device, and said set offiducial markers are imaged during operation of said optical source. 2.The surgical guidance system according to claim 1 wherein said opticalprojection device and said optical source emit optical radiation withoverlapping spectra.
 3. The surgical guidance system according to claim1 wherein at least two of said two or more cameras are employed for bothimaging the backscattered optical radiation and imaging said set offiducial markers.
 4. The surgical guidance system according to claim 3wherein each camera that is employed for both imaging the backscatteredoptical radiation and imaging said set of fiducial markers has arespective optical filter associated therewith, wherein said respectiveoptical filter is configured to selectively transmit both thebackscattered optical radiation and optical radiation received from saidset of fiducial markers upon illumination with said optical source. 5.The surgical guidance system according to claim 1 wherein said opticalprojection device and said optical source emit optical radiation withthe infrared.
 6. The surgical guidance system according to claim 1wherein said optical projection device, said optical source, and saidtwo or more cameras are rigidly mounted on a frame, such that a fixedspatial relationship exists therebetween, thereby maintaining a fixedcalibration of said system without requiring inter-operativerecalibration.
 7. The surgical guidance system according to claim 1wherein said optical projection device is a structured light projectiondevice.
 8. The surgical guidance system according to claim 1 whereinsaid optical projection device and said one or more first cameras areconfigured for obtaining topological image data corresponding to two ormore spectral bands.
 9. A surgical guidance system comprising: a storagemedium for storing pre-operative image data associated with a patient; asurface topology imaging and optical position measurement systemcomprising: an optical projection device configured to project opticalradiation onto an exposed surface of the patient, such thatbackscattered optical radiation is suitable for optical surface topologydetection; two or more cameras, wherein at least one of said two or morecameras is configured for imaging the backscattered optical radiation,and wherein at least two of said two or more cameras are configured forimaging a set of fiducial markers provided on a movable instrument; anda surgical guidance controller operatively connected to said surfacetopology imaging and optical position measurement system and saidstorage medium, wherein said surgical guidance controller includes aprocessor configured to: control said optical projection device toilluminate the exposed surface and obtain, based on one or more imagesfrom said at least one camera, topological image data associated withthe exposed surface; and obtain, from said two or more cameras,positional image data associated with said set of fiducial markers;determine a position and an orientation of the movable instrumentrelative to the exposed surface; and register said topological imagedata, and said position and orientation of the movable instrument tosaid pre-operative image data; wherein at least one camera has aplurality of optical filters associated therewith, said plurality ofoptical filters including a first optical filter suitable forselectively transmitting the backscattered optical radiation and asecond optical filter suitable for selectively transmitting opticalradiation received from said set of fiducial markers; and wherein saidfirst optical filter and said second optical filter are removablypositionable in front of said at least one camera under control of saidsurgical guidance controller, such that the imaging the backscatteredoptical radiation is temporally interleaved with the imaging said set offiducial markers.
 10. The surgical guidance system according to claim 9wherein said set of fiducial markers are a set of passive fiducialmarkers, said system further comprising an optical source having awavelength selected to illuminate the set of passive fiducial markers.11. The surgical guidance system according to claim 10 wherein saidoptical projection device and said optical source emit optical radiationwith the infrared.
 12. The surgical guidance system according to claim 9wherein said set of fiducial markers are a set of active fiducialmarkers.
 13. The surgical guidance system according to claim 12 whereinoptical radiation emitted by said optical projection device and opticalradiation emitted by the said set of active fiducial markers is in theinfrared.
 14. The surgical guidance system according to claim 9 whereinsaid plurality of optical filters comprises at least one additionaloptical filter configured for selectively transmitting the backscatteredoptical radiation in an additional spectral band.
 15. The surgicalguidance system according to claim 9 wherein said optical projectiondevice and said two or more cameras are rigidly mounted on a frame, suchthat a fixed spatial relationship exists therebetween, therebymaintaining a fixed calibration of said system without requiringinter-operative recalibration.
 16. The surgical guidance systemaccording to claim 9 wherein said optical projection device is astructured light projection device.
 17. A surgical guidance systemcomprising: a storage medium for storing pre-operative image dataassociated with a patient; a surface topology imaging and opticalposition measurement system comprising: an optical projection deviceconfigured to project optical radiation onto an exposed surface of thepatient, such that backscattered optical radiation is suitable foroptical surface topology detection; two or more cameras, wherein atleast one of said two or more cameras is configured for imaging thebackscattered optical radiation, and wherein at least two of said two ormore cameras are configured for imaging a set of fiducial markersprovided on a movable instrument; and a surgical guidance controlleroperatively connected to said surface topology imaging and opticalposition measurement system and said storage medium, wherein saidsurgical guidance controller includes a processor configured to: controlsaid optical projection device to illuminate the exposed surface andobtain, based on one or more images from said at least one camera,topological image data associated with the exposed surface; and obtain,from said two or more cameras, positional image data associated withsaid set of fiducial markers; determine a position and an orientation ofthe movable instrument relative to the exposed surface; and registersaid topological image data, and said position and orientation of themovable instrument to said pre-operative image data; wherein saidsurgical guidance controller is configured to control said surfacetopology imaging and optical position measurement system such that theimaging of the backscattered optical radiation is temporally interleavedwith the imaging of said set of fiducial markers.
 18. The surgicalguidance system according to claim 17 wherein said set of fiducialmarkers are a set of passive fiducial markers, said system furthercomprising an optical source having a wavelength selected to illuminatethe set of passive fiducial markers.
 19. The surgical guidance systemaccording to claim 17 wherein said set of fiducial markers are a set ofactive fiducial markers.
 20. The surgical guidance system according toclaim 17 wherein said optical projection device is a structured lightprojection device.